Provable Smoothness Guarantees for Black-Box Variational Inference
This provides theoretical foundations for improving convergence in variational inference, which is incremental but important for practitioners in Bayesian machine learning.
The paper tackles the problem of establishing provable convergence guarantees for black-box variational inference by showing that for location-scale family approximations, if the target distribution is M-Lipschitz smooth, then the objective function (excluding entropy) inherits this smoothness property, enabling rigorous analysis of optimization behavior.
Black-box variational inference tries to approximate a complex target distribution though a gradient-based optimization of the parameters of a simpler distribution. Provable convergence guarantees require structural properties of the objective. This paper shows that for location-scale family approximations, if the target is M-Lipschitz smooth, then so is the objective, if the entropy is excluded. The key proof idea is to describe gradients in a certain inner-product space, thus permitting use of Bessel's inequality. This result gives insight into how to parameterize distributions, gives bounds the location of the optimal parameters, and is a key ingredient for convergence guarantees.