On the detrimental effect of invariances in the likelihood for variational inference
This work addresses a fundamental issue in variational inference for machine learning practitioners, particularly in Bayesian neural networks, by identifying invariances as a key cause of underfitting, though it is incremental as it builds on prior analyses without presenting a full solution.
The paper tackles the underfitting problem in variational Bayesian posterior inference for over-parametrized models like Bayesian neural networks, showing that invariances in the likelihood function create complex posterior structures with discrete or continuous modes that Gaussian mean-field approximations poorly approximate, resulting in an additional gap in the evidence lower bound that vanishes as the approximation reverts to the prior.
Variational Bayesian posterior inference often requires simplifying approximations such as mean-field parametrisation to ensure tractability. However, prior work has associated the variational mean-field approximation for Bayesian neural networks with underfitting in the case of small datasets or large model sizes. In this work, we show that invariances in the likelihood function of over-parametrised models contribute to this phenomenon because these invariances complicate the structure of the posterior by introducing discrete and/or continuous modes which cannot be well approximated by Gaussian mean-field distributions. In particular, we show that the mean-field approximation has an additional gap in the evidence lower bound compared to a purpose-built posterior that takes into account the known invariances. Importantly, this invariance gap is not constant; it vanishes as the approximation reverts to the prior. We proceed by first considering translation invariances in a linear model with a single data point in detail. We show that, while the true posterior can be constructed from a mean-field parametrisation, this is achieved only if the objective function takes into account the invariance gap. Then, we transfer our analysis of the linear model to neural networks. Our analysis provides a framework for future work to explore solutions to the invariance problem.