Understanding Stochastic Natural Gradient Variational Inference
This provides theoretical grounding for a widely used inference method, though it is incremental as it clarifies existing limitations.
The paper tackled the lack of non-asymptotic convergence rate analysis for stochastic natural gradient variational inference (NGVI), proving an O(1/T) rate for conjugate likelihoods and showing that for non-conjugate likelihoods, global convergence is unlikely without new insights.
Stochastic natural gradient variational inference (NGVI) is a popular posterior inference method with applications in various probabilistic models. Despite its wide usage, little is known about the non-asymptotic convergence rate in the \emph{stochastic} setting. We aim to lessen this gap and provide a better understanding. For conjugate likelihoods, we prove the first $\mathcal{O}(\frac{1}{T})$ non-asymptotic convergence rate of stochastic NGVI. The complexity is no worse than stochastic gradient descent (\aka black-box variational inference) and the rate likely has better constant dependency that leads to faster convergence in practice. For non-conjugate likelihoods, we show that stochastic NGVI with the canonical parameterization implicitly optimizes a non-convex objective. Thus, a global convergence rate of $\mathcal{O}(\frac{1}{T})$ is unlikely without some significant new understanding of optimizing the ELBO using natural gradients.