Surrogate Likelihoods for Variational Annealed Importance Sampling
This work addresses a limitation in hybrid Bayesian inference methods for researchers and practitioners, offering an incremental improvement by enabling data subsampling.
The paper tackles the challenge of supporting data subsampling in hybrid variational-MCMC methods by introducing a surrogate likelihood that can be learned jointly with variational parameters, enabling a trade-off between inference fidelity and computational cost. In empirical comparisons, the method performs well and is suitable for black-box inference in probabilistic programming frameworks.
Variational inference is a powerful paradigm for approximate Bayesian inference with a number of appealing properties, including support for model learning and data subsampling. By contrast MCMC methods like Hamiltonian Monte Carlo do not share these properties but remain attractive since, contrary to parametric methods, MCMC is asymptotically unbiased. For these reasons researchers have sought to combine the strengths of both classes of algorithms, with recent approaches coming closer to realizing this vision in practice. However, supporting data subsampling in these hybrid methods can be a challenge, a shortcoming that we address by introducing a surrogate likelihood that can be learned jointly with other variational parameters. We argue theoretically that the resulting algorithm permits the user to make an intuitive trade-off between inference fidelity and computational cost. In an extensive empirical comparison we show that our method performs well in practice and that it is well-suited for black-box inference in probabilistic programming frameworks.