Bayesian Neural Network Ensembles
This work addresses the need for a principled Bayesian approach to uncertainty estimation in neural networks, offering a scalable alternative to costly Bayesian neural networks.
The paper tackles the problem of estimating predictive uncertainty in neural networks by proposing a modified ensembling scheme that provides a consistent estimator of the Bayesian posterior in wide neural networks, regularizing parameters about values drawn from a prior distribution.
Ensembles of neural networks (NNs) have long been used to estimate predictive uncertainty; a small number of NNs are trained from different initialisations and sometimes on differing versions of the dataset. The variance of the ensemble's predictions is interpreted as its epistemic uncertainty. The appeal of ensembling stems from being a collection of regular NNs - this makes them both scalable and easily implementable. They have achieved strong empirical results in recent years, often presented as a practical alternative to more costly Bayesian NNs (BNNs). The departure from Bayesian methodology is of concern since the Bayesian framework provides a principled, widely-accepted approach to handling uncertainty. In this extended abstract we derive and implement a modified NN ensembling scheme, which provides a consistent estimator of the Bayesian posterior in wide NNs - regularising parameters about values drawn from a prior distribution.