VISA: Variational Inference with Sequential Sample-Average Approximations
This addresses computational efficiency for researchers using variational inference in simulation-based models, but it is incremental as it builds on existing methods with specific optimizations.
The paper tackles the problem of high computational cost in approximate inference for intensive models like numerical simulations, and presents VISA, a method that achieves comparable accuracy to standard importance-weighted forward-KL variational inference with computational savings of a factor two or more.
We present variational inference with sequential sample-average approximation (VISA), a method for approximate inference in computationally intensive models, such as those based on numerical simulations. VISA extends importance-weighted forward-KL variational inference by employing a sequence of sample-average approximations, which are considered valid inside a trust region. This makes it possible to reuse model evaluations across multiple gradient steps, thereby reducing computational cost. We perform experiments on high-dimensional Gaussians, Lotka-Volterra dynamics, and a Pickover attractor, which demonstrate that VISA can achieve comparable approximation accuracy to standard importance-weighted forward-KL variational inference with computational savings of a factor two or more for conservatively chosen learning rates.