DEAM: Adaptive Momentum with Discriminative Weight for Stochastic Optimization
This work addresses optimization challenges for deep learning practitioners by proposing an incremental improvement to momentum-based algorithms.
The paper tackles the problem of error propagation in momentum-based optimization algorithms like ADAM by introducing DEAM, which automatically computes momentum weight based on a discriminative angle and includes a backtrack term to restrict redundant updates, achieving faster convergence rates in training deep learning models for both convex and non-convex situations.
Optimization algorithms with momentum, e.g., (ADAM), have been widely used for building deep learning models due to the faster convergence rates compared with stochastic gradient descent (SGD). Momentum helps accelerate SGD in the relevant directions in parameter updating, which can minify the oscillations of parameters update route. However, there exist errors in some update steps in optimization algorithms with momentum like ADAM. The fixed momentum weight (e.g., β_1 in ADAM) will propagate errors in momentum computing. In this paper, we introduce a novel optimization algorithm, namely Discriminative wEight on Adaptive Momentum (DEAM). Instead of assigning the momentum term weight with a fixed hyperparameter, DEAM proposes to compute the momentum weight automatically based on the discriminative angle. In this way, DEAM involves fewer hyperparameters. DEAM also contains a novel backtrack term, which restricts redundant updates when the correction of the last step is needed. Extensive experiments demonstrate that DEAM can achieve a faster convergence rate than the existing optimization algorithms in training the deep learning models of both convex and non-convex situations.