Effective continuous equations for adaptive SGD: a stochastic analysis view
This provides incremental theoretical insights for researchers in optimization and machine learning, focusing on understanding adaptive SGD dynamics.
The paper tackles the theoretical analysis of adaptive SGD methods by deriving effective continuous stochastic dynamics using the stochastic modified equations framework, showing that sampling-induced noise manifests as independent Brownian motions in the limit and characterizing scaling rules between learning rate and hyperparameters.
We present a theoretical analysis of some popular adaptive Stochastic Gradient Descent (SGD) methods in the small learning rate regime. Using the stochastic modified equations framework introduced by Li et al., we derive effective continuous stochastic dynamics for these methods. Our key contribution is that sampling-induced noise in SGD manifests in the limit as independent Brownian motions driving the parameter and gradient second momentum evolutions. Furthermore, extending the approach of Malladi et al., we investigate scaling rules between the learning rate and key hyperparameters in adaptive methods, characterising all non-trivial limiting dynamics.