Approximate Inference with the Variational Holder Bound
This addresses the problem of non-convex optimization in variational inference for researchers and practitioners in machine learning, though it appears incremental as it builds on existing variational methods.
The paper tackles approximate Bayesian inference by introducing the Variational Holder (VH) bound as an alternative to Variational Bayes (VB), resulting in a convex optimization problem that can be solved with standard algorithms and analyzed for error.
We introduce the Variational Holder (VH) bound as an alternative to Variational Bayes (VB) for approximate Bayesian inference. Unlike VB which typically involves maximization of a non-convex lower bound with respect to the variational parameters, the VH bound involves minimization of a convex upper bound to the intractable integral with respect to the variational parameters. Minimization of the VH bound is a convex optimization problem; hence the VH method can be applied using off-the-shelf convex optimization algorithms and the approximation error of the VH bound can also be analyzed using tools from convex optimization literature. We present experiments on the task of integrating a truncated multivariate Gaussian distribution and compare our method to VB, EP and a state-of-the-art numerical integration method for this problem.