Dual Averaging is Surprisingly Effective for Deep Learning Optimization
This addresses the ad-hoc optimizer selection issue for deep learning practitioners, offering a simpler, non-adaptive alternative that works across domains, though it is incremental as it builds on existing dual averaging ideas.
The paper tackles the problem of optimizer choice in deep learning, where using the wrong method can degrade performance, and proposes Modernized Dual Averaging (MDA), which matches SGD+M in computer vision and Adam in NLP tasks.
First-order stochastic optimization methods are currently the most widely used class of methods for training deep neural networks. However, the choice of the optimizer has become an ad-hoc rule that can significantly affect the performance. For instance, SGD with momentum (SGD+M) is typically used in computer vision (CV) and Adam is used for training transformer models for Natural Language Processing (NLP). Using the wrong method can lead to significant performance degradation. Inspired by the dual averaging algorithm, we propose Modernized Dual Averaging (MDA), an optimizer that is able to perform as well as SGD+M in CV and as Adam in NLP. Our method is not adaptive and is significantly simpler than Adam. We show that MDA induces a decaying uncentered $L_2$-regularization compared to vanilla SGD+M and hypothesize that this may explain why it works on NLP problems where SGD+M fails.