'In-Between' Uncertainty in Bayesian Neural Networks
This addresses a critical issue for active learning, Bayesian optimization, and out-of-distribution robustness, though it is incremental as it builds on existing methods.
The paper identifies a limitation in mean-field variational inference (MFVI) for Bayesian neural networks, where it fails to provide calibrated uncertainty estimates between separated data regions, leading to overconfident predictions on out-of-distribution data. It shows that the linearised Laplace approximation improves this 'in-between' uncertainty for small networks.
We describe a limitation in the expressiveness of the predictive uncertainty estimate given by mean-field variational inference (MFVI), a popular approximate inference method for Bayesian neural networks. In particular, MFVI fails to give calibrated uncertainty estimates in between separated regions of observations. This can lead to catastrophically overconfident predictions when testing on out-of-distribution data. Avoiding such overconfidence is critical for active learning, Bayesian optimisation and out-of-distribution robustness. We instead find that a classical technique, the linearised Laplace approximation, can handle 'in-between' uncertainty much better for small network architectures.