Adaptive Methods through the Lens of SDEs: Theoretical Insights on the Role of Noise
This work provides theoretical insights into adaptive optimization methods, which could inform best training practices and scaling rules for deep learning practitioners, though it is incremental in building on existing SDE analyses.
The authors tackled the incomplete theoretical understanding of adaptive optimization methods in deep learning by introducing novel stochastic differential equations (SDEs) for SignSGD, RMSprop(W), and Adam(W), which accurately describe these optimizers and reveal relationships between adaptivity, gradient noise, and curvature, supported by experimental verification on various neural network architectures.
Despite the vast empirical evidence supporting the efficacy of adaptive optimization methods in deep learning, their theoretical understanding is far from complete. This work introduces novel SDEs for commonly used adaptive optimizers: SignSGD, RMSprop(W), and Adam(W). These SDEs offer a quantitatively accurate description of these optimizers and help illuminate an intricate relationship between adaptivity, gradient noise, and curvature. Our novel analysis of SignSGD highlights a noteworthy and precise contrast to SGD in terms of convergence speed, stationary distribution, and robustness to heavy-tail noise. We extend this analysis to AdamW and RMSpropW, for which we observe that the role of noise is much more complex. Crucially, we support our theoretical analysis with experimental evidence by verifying our insights: this includes numerically integrating our SDEs using Euler-Maruyama discretization on various neural network architectures such as MLPs, CNNs, ResNets, and Transformers. Our SDEs accurately track the behavior of the respective optimizers, especially when compared to previous SDEs derived for Adam and RMSprop. We believe our approach can provide valuable insights into best training practices and novel scaling rules.