Bootstrap Your Own Variance
This work addresses uncertainty estimation for machine learning models, particularly in self-supervised learning, but it appears incremental as it builds on existing methods.
The paper tackles the problem of model uncertainty estimation by proposing Bootstrap Your Own Variance (BYOV), which combines Bootstrap Your Own Latent (BYOL) with Bayes by Backprop (BBB), resulting in improved calibration and reliability metrics, such as a +2.83% test ECE and +1.03% test Brier score over the deterministic baseline.
Understanding model uncertainty is important for many applications. We propose Bootstrap Your Own Variance (BYOV), combining Bootstrap Your Own Latent (BYOL), a negative-free Self-Supervised Learning (SSL) algorithm, with Bayes by Backprop (BBB), a Bayesian method for estimating model posteriors. We find that the learned predictive std of BYOV vs. a supervised BBB model is well captured by a Gaussian distribution, providing preliminary evidence that the learned parameter posterior is useful for label free uncertainty estimation. BYOV improves upon the deterministic BYOL baseline (+2.83% test ECE, +1.03% test Brier) and presents better calibration and reliability when tested with various augmentations (eg: +2.4% test ECE, +1.2% test Brier for Salt & Pepper noise).