The Variational Gaussian Process
This work addresses the problem of scalable and flexible approximate inference for machine learning practitioners, offering a novel method that is not incremental but introduces a new paradigm for variational families.
The paper tackles the challenge of approximating complex posterior distributions in Bayesian inference by introducing the variational Gaussian process (VGP), a nonparametric variational family that adapts its shape to match these distributions, achieving new state-of-the-art results in unsupervised learning tasks like deep latent Gaussian models and DRAW.
Variational inference is a powerful tool for approximate inference, and it has been recently applied for representation learning with deep generative models. We develop the variational Gaussian process (VGP), a Bayesian nonparametric variational family, which adapts its shape to match complex posterior distributions. The VGP generates approximate posterior samples by generating latent inputs and warping them through random non-linear mappings; the distribution over random mappings is learned during inference, enabling the transformed outputs to adapt to varying complexity. We prove a universal approximation theorem for the VGP, demonstrating its representative power for learning any model. For inference we present a variational objective inspired by auto-encoders and perform black box inference over a wide class of models. The VGP achieves new state-of-the-art results for unsupervised learning, inferring models such as the deep latent Gaussian model and the recently proposed DRAW.