Rapidly Adapting Moment Estimation
This work addresses the optimization of deep neural networks for machine learning practitioners, presenting an incremental improvement over existing adaptive gradient methods.
The paper tackled the problem of training deep neural networks by proposing a new adaptive gradient method, RAME, which uses the first moment of gradients to compute learning rates, resulting in faster convergence and better generalization compared to existing methods like Adam and RMSprop.
Adaptive gradient methods such as Adam have been shown to be very effective for training deep neural networks (DNNs) by tracking the second moment of gradients to compute the individual learning rates. Differently from existing methods, we make use of the most recent first moment of gradients to compute the individual learning rates per iteration. The motivation behind it is that the dynamic variation of the first moment of gradients may provide useful information to obtain the learning rates. We refer to the new method as the rapidly adapting moment estimation (RAME). The theoretical convergence of deterministic RAME is studied by using an analysis similar to the one used in [1] for Adam. Experimental results for training a number of DNNs show promising performance of RAME w.r.t. the convergence speed and generalization performance compared to the stochastic heavy-ball (SHB) method, Adam, and RMSprop.