Dynamic Momentum Recalibration in Online Gradient Learning
This work addresses the suboptimal parameter updates caused by fixed momentum coefficients in deep learning optimizers, which is a problem for practitioners seeking more efficient and effective training.
This paper reinterprets gradient updates in deep learning optimizers through signal processing, revealing that fixed momentum coefficients distort the bias-variance balance. The authors propose SGDF, an optimizer that dynamically recalibrates momentum by computing an online, time-varying gain to minimize mean-squared error in gradient estimation, achieving performance on par with or surpassing state-of-the-art optimizers.
Stochastic Gradient Descent (SGD) and its momentum variants form the backbone of deep learning optimization, yet the underlying dynamics of their gradient behavior remain insufficiently understood. In this work, we reinterpret gradient updates through the lens of signal processing and reveal that fixed momentum coefficients inherently distort the balance between bias and variance, leading to skewed or suboptimal parameter updates. To address this, we propose SGDF (SGD with Filter), an optimizer inspired by the principles of Optimal Linear Filtering. SGDF computes an online, time-varying gain to dynamically refine gradient estimation by minimizing the mean-squared error, thereby achieving an optimal trade-off between noise suppression and signal preservation. Furthermore, our approach could extend to other optimizers, showcasing its broad applicability to optimization frameworks. Extensive experiments across diverse architectures and benchmarks demonstrate SGDF surpasses conventional momentum methods and achieves performance on par with or surpassing state-of-the-art optimizers.