Gaussian Mean Field Variational Inference can Overestimate Predictive Variance
This work corrects a fundamental misunderstanding about MFVI for Bayesian deep learning practitioners, revealing a previously unrecognized failure mode.
The paper shows that Mean Field Variational Inference (MFVI) can overestimate predictive variance in directions where training data concentrates, contrary to the common belief that it always underestimates variance. For test points from the training distribution, MFVI's expected predictive variance exceeds that of the exact posterior, and in extreme cases, it fails to reduce variance compared to the prior.
Mean Field Variational Inference (MFVI) is widely understood to underestimate posterior variance. By analysing conjugate Bayesian Linear Regression (BLR), we show that this characterization is incomplete: while MFVI underestimates the variance in parameter space, it can overestimate the predictive variance compared to the exact posterior. We show that if the MFVI posterior underestimates predictive variances in some directions, it necessarily overestimates them in others. Crucially, this overestimation occurs in directions where the training data concentrates. This leads to the surprising result that, for a test point drawn from the training distribution, MFVI's expected predictive variance exceeds that of the exact posterior. We demonstrate a pathological case of this effect, where the MFVI posterior fails to reduce predictive variance compared to the prior on in distribution data. We connect these results to the Cold Posterior Effect, arguing that varying the temperature can correct this overestimation, yielding predictions closer to those of the exact posterior. We validate our theory on synthetic and real-world regression tasks.