You Only Accept Samples Once: Fast, Self-Correcting Stochastic Variational Inference
This addresses the problem of slow convergence in variational inference for practitioners working with large Bayesian models, though it appears incremental as it builds on existing stochastic optimization frameworks.
The paper tackles the computational inefficiency of stochastic variational inference in large Bayesian hierarchical models by introducing YOASOVI, an algorithm that uses acceptance sampling instead of Monte Carlo sampling to draw only one sample per iteration. Empirical results show YOASOVI converges faster in clock time and achieves better optimal neighborhoods than existing methods like regularized Monte Carlo and Quasi-Monte Carlo VI.
We introduce YOASOVI, an algorithm for performing fast, self-correcting stochastic optimization for Variational Inference (VI) on large Bayesian heirarchical models. To accomplish this, we take advantage of available information on the objective function used for stochastic VI at each iteration and replace regular Monte Carlo sampling with acceptance sampling. Rather than spend computational resources drawing and evaluating over a large sample for the gradient, we draw only one sample and accept it with probability proportional to the expected improvement in the objective. The following paper develops two versions of the algorithm: the first one based on a naive intuition, and another building up the algorithm as a Metropolis-type scheme. Empirical results based on simulations and benchmark datasets for multivariate Gaussian mixture models show that YOASOVI consistently converges faster (in clock time) and within better optimal neighborhoods than both regularized Monte Carlo and Quasi-Monte Carlo VI algorithms.