Vprop: Variational Inference using RMSprop
This provides an easy-to-implement solution for practitioners in Bayesian deep learning, though it is incremental as it builds on existing optimization techniques.
The paper tackles the challenge of implementing Bayesian deep learning methods by proposing Vprop, a Gaussian variational inference method that requires only two minor changes to the standard RMSprop optimizer and reduces memory usage by half compared to Black-Box Variational Inference.
Many computationally-efficient methods for Bayesian deep learning rely on continuous optimization algorithms, but the implementation of these methods requires significant changes to existing code-bases. In this paper, we propose Vprop, a method for Gaussian variational inference that can be implemented with two minor changes to the off-the-shelf RMSprop optimizer. Vprop also reduces the memory requirements of Black-Box Variational Inference by half. We derive Vprop using the conjugate-computation variational inference method, and establish its connections to Newton's method, natural-gradient methods, and extended Kalman filters. Overall, this paper presents Vprop as a principled, computationally-efficient, and easy-to-implement method for Bayesian deep learning.