Posterior Refinement Improves Sample Efficiency in Bayesian Neural Networks
This addresses sample efficiency issues in Bayesian deep learning, offering a simple post hoc method for practitioners, though it is incremental as it builds on existing parametric approximations.
The paper tackles the problem of poor predictive performance in Bayesian neural networks due to posterior approximation errors, showing that refining Gaussian approximate posteriors with normalizing flows yields competitive results with full-batch Hamiltonian Monte Carlo.
Monte Carlo (MC) integration is the de facto method for approximating the predictive distribution of Bayesian neural networks (BNNs). But, even with many MC samples, Gaussian-based BNNs could still yield bad predictive performance due to the posterior approximation's error. Meanwhile, alternatives to MC integration tend to be more expensive or biased. In this work, we experimentally show that the key to good MC-approximated predictive distributions is the quality of the approximate posterior itself. However, previous methods for obtaining accurate posterior approximations are expensive and non-trivial to implement. We, therefore, propose to refine Gaussian approximate posteriors with normalizing flows. When applied to last-layer BNNs, it yields a simple \emph{post hoc} method for improving pre-existing parametric approximations. We show that the resulting posterior approximation is competitive with even the gold-standard full-batch Hamiltonian Monte Carlo.