SVRG and Beyond via Posterior Correction
This work addresses the problem of accelerating deep learning training for researchers and practitioners by providing incremental improvements to SVRG through a Bayesian framework.
The paper tackles the limited success of Stochastic Variance Reduced Gradient (SVRG) in deep learning by connecting it to Bayesian posterior correction, showing SVRG as a special case and deriving new variants like a Newton-like method and an Adam-like extension that improves Transformer language model training.
Stochastic Variance Reduced Gradient (SVRG) and its variants aim to speed-up training by using gradient corrections, but have seen limited success in deep learning. Here, we show surprising new foundational connections of SVRG to a recently proposed Bayesian method called posterior correction. Specifically, we show that SVRG is recovered as a special case of posterior correction over the isotropic-Gaussian family, while novel extensions are automatically obtained by using more flexible exponential families. We derive two new SVRG variants by using Gaussian families: First, a Newton-like variant that employs novel Hessian corrections, and second, an Adam-like extension that improves pretraining and finetuning of Transformer language models. This is the first work to connect SVRG to Bayes and use it to boost variational training for deep networks.