Incorporating Unlabelled Data into Bayesian Neural Networks
This work addresses a problem for machine learning practitioners by enabling BNNs to leverage unlabelled data, though it appears incremental as it builds on existing contrastive and variational methods.
The paper tackles the limitation of Bayesian Neural Networks (BNNs) in utilizing unlabelled data by introducing Self-Supervised Bayesian Neural Networks, which improve predictive performance, particularly in low-budget regimes, by learning better prior predictive distributions through contrastive pretraining.
Conventional Bayesian Neural Networks (BNNs) are unable to leverage unlabelled data to improve their predictions. To overcome this limitation, we introduce Self-Supervised Bayesian Neural Networks, which use unlabelled data to learn models with suitable prior predictive distributions. This is achieved by leveraging contrastive pretraining techniques and optimising a variational lower bound. We then show that the prior predictive distributions of self-supervised BNNs capture problem semantics better than conventional BNN priors. In turn, our approach offers improved predictive performance over conventional BNNs, especially in low-budget regimes.