Covariance Matrices and Influence Scores for Mean Field Variational Bayes
This addresses a major limitation in variational inference for practitioners needing reliable uncertainty estimates in large-scale Bayesian models, though it is incremental as it builds on existing MFVB frameworks.
The authors tackled the problem of Mean Field Variational Bayes (MFVB) underestimating uncertainty and lacking covariance information by developing Linear Response Variational Bayes (LRVB), a fast method that provides accurate uncertainty estimates and influence scores for exponential families, demonstrating its accuracy and scalability on Gaussian mixture models with simulated and real data.
Mean field variational Bayes (MFVB) is a popular posterior approximation method due to its fast runtime on large-scale data sets. However, it is well known that a major failing of MFVB is that it underestimates the uncertainty of model variables (sometimes severely) and provides no information about model variable covariance. We develop a fast, general methodology for exponential families that augments MFVB to deliver accurate uncertainty estimates for model variables -- both for individual variables and coherently across variables. MFVB for exponential families defines a fixed-point equation in the means of the approximating posterior, and our approach yields a covariance estimate by perturbing this fixed point. Inspired by linear response theory, we call our method linear response variational Bayes (LRVB). We also show how LRVB can be used to quickly calculate a measure of the influence of individual data points on parameter point estimates. We demonstrate the accuracy and scalability of our method by learning Gaussian mixture models for both simulated and real data.