Covariance Matrices for Mean Field Variational Bayes
This addresses a major failing in MFVB for researchers and practitioners needing reliable uncertainty estimates in large-scale Bayesian inference, though it is an incremental improvement focused on enhancing existing methods.
The paper tackles the problem of Mean Field Variational Bayes (MFVB) underestimating uncertainty and lacking covariance information by developing a fast method called linear response variational Bayes (LRVB) that augments MFVB to provide accurate uncertainty estimates for model variables, demonstrating its accuracy on simulated data sets.
Mean Field Variational Bayes (MFVB) is a popular posterior approximation method due to its fast runtime on large-scale data sets. However, it is well known that a major failing of MFVB is its (sometimes severe) underestimates of the uncertainty of model variables and lack of information about model variable covariance. We develop a fast, general methodology for exponential families that augments MFVB to deliver accurate uncertainty estimates for model variables -- both for individual variables and coherently across variables. MFVB for exponential families defines a fixed-point equation in the means of the approximating posterior, and our approach yields a covariance estimate by perturbing this fixed point. Inspired by linear response theory, we call our method linear response variational Bayes (LRVB). We demonstrate the accuracy of our method on simulated data sets.