Gaussian Mean Field Regularizes by Limiting Learned Information
This work addresses a fundamental issue in machine learning for researchers and practitioners using variational inference, providing theoretical insights into regularization mechanisms.
The paper tackles the problem of understanding the regularizing effect of Gaussian mean field variational inference by showing it improves generalization by limiting mutual information between parameters and data through noise, quantifying a maximum capacity and connecting it to generalization error, with experiments demonstrating effective regularization on supervised and unsupervised tasks.
Variational inference with a factorized Gaussian posterior estimate is a widely used approach for learning parameters and hidden variables. Empirically, a regularizing effect can be observed that is poorly understood. In this work, we show how mean field inference improves generalization by limiting mutual information between learned parameters and the data through noise. We quantify a maximum capacity when the posterior variance is either fixed or learned and connect it to generalization error, even when the KL-divergence in the objective is rescaled. Our experiments demonstrate that bounding information between parameters and data effectively regularizes neural networks on both supervised and unsupervised tasks.