Variational Laplace for Bayesian neural networks
This work addresses the computational bottleneck in variational inference for Bayesian neural networks, offering a more efficient method for practitioners, though it is incremental as it builds on existing variational frameworks.
The paper tackled the challenge of estimating the evidence lower bound (ELBO) in Bayesian neural networks without stochastic weight sampling by developing Variational Laplace, which uses a local curvature approximation. The result showed improved test performance and expected calibration errors compared to maximum a-posteriori inference and standard variational inference, with specific gains in efficiency and convergence.
We develop variational Laplace for Bayesian neural networks (BNNs) which exploits a local approximation of the curvature of the likelihood to estimate the ELBO without the need for stochastic sampling of the neural-network weights. The Variational Laplace objective is simple to evaluate, as it is (in essence) the log-likelihood, plus weight-decay, plus a squared-gradient regularizer. Variational Laplace gave better test performance and expected calibration errors than maximum a-posteriori inference and standard sampling-based variational inference, despite using the same variational approximate posterior. Finally, we emphasise care needed in benchmarking standard VI as there is a risk of stopping before the variance parameters have converged. We show that early-stopping can be avoided by increasing the learning rate for the variance parameters.