Fast and Simple Natural-Gradient Variational Inference with Mixture of Exponential-family Approximations
This work incrementally expands the applicability of natural gradients in Bayesian inference, making them more useful for practitioners dealing with multimodal posterior distributions.
The paper tackles the challenge of applying natural-gradient methods to structured variational approximations like mixtures of exponential-family distributions, which can model complex posteriors, and demonstrates faster convergence compared to gradient-based methods.
Natural-gradient methods enable fast and simple algorithms for variational inference, but due to computational difficulties, their use is mostly limited to \emph{minimal} exponential-family (EF) approximations. In this paper, we extend their application to estimate \emph{structured} approximations such as mixtures of EF distributions. Such approximations can fit complex, multimodal posterior distributions and are generally more accurate than unimodal EF approximations. By using a \emph{minimal conditional-EF} representation of such approximations, we derive simple natural-gradient updates. Our empirical results demonstrate a faster convergence of our natural-gradient method compared to black-box gradient-based methods with reparameterization gradients. Our work expands the scope of natural gradients for Bayesian inference and makes them more widely applicable than before.