Variational Laplace for Bayesian neural networks
This work provides an improved and more efficient method for Bayesian neural network inference, which is beneficial for researchers and practitioners working with uncertainty quantification in deep learning.
The authors developed Variational Laplace for Bayesian neural networks (BNNs) to estimate the ELBO without stochastic sampling of weights. This method achieved better test performance and expected calibration errors compared to maximum a-posteriori inference and standard sampling-based variational inference.
We develop variational Laplace for Bayesian neural networks (BNNs) which exploits a local approximation of the curvature of the likelihood to estimate the ELBO without the need for stochastic sampling of the neural-network weights. The Variational Laplace objective is simple to evaluate, as it is (in essence) the log-likelihood, plus weight-decay, plus a squared-gradient regularizer. Variational Laplace gave better test performance and expected calibration errors than maximum a-posteriori inference and standard sampling-based variational inference, despite using the same variational approximate posterior. Finally, we emphasise care needed in benchmarking standard VI as there is a risk of stopping before the variance parameters have converged. We show that early-stopping can be avoided by increasing the learning rate for the variance parameters.