Practical and Matching Gradient Variance Bounds for Black-Box Variational Bayesian Inference
This work addresses a theoretical gap in BBVI convergence analysis for researchers in machine learning and statistics, but it is incremental as it builds on existing SGD conditions.
The paper tackles the problem of understanding gradient variance in black-box variational inference (BBVI) by showing that BBVI satisfies a matching bound corresponding to the ABC condition used in SGD convergence studies for smooth and quadratically-growing log-likelihoods, and it generalizes to nonlinear covariance parameterizations while proving superior dimensional dependence for mean-field parameterization.
Understanding the gradient variance of black-box variational inference (BBVI) is a crucial step for establishing its convergence and developing algorithmic improvements. However, existing studies have yet to show that the gradient variance of BBVI satisfies the conditions used to study the convergence of stochastic gradient descent (SGD), the workhorse of BBVI. In this work, we show that BBVI satisfies a matching bound corresponding to the $ABC$ condition used in the SGD literature when applied to smooth and quadratically-growing log-likelihoods. Our results generalize to nonlinear covariance parameterizations widely used in the practice of BBVI. Furthermore, we show that the variance of the mean-field parameterization has provably superior dimensional dependence.