An Optimistic Acceleration of AMSGrad for Nonconvex Optimization
This work addresses optimization efficiency for deep learning practitioners, but it is incremental as it builds upon the widely used AMSGrad algorithm.
The authors tackled the problem of accelerating convergence in nonconvex optimization for deep neural networks by proposing a new variant of AMSGrad that leverages gradient predictability and optimistic online learning, resulting in practical speedup demonstrated in numerical experiments.
We propose a new variant of AMSGrad, a popular adaptive gradient based optimization algorithm widely used for training deep neural networks. Our algorithm adds prior knowledge about the sequence of consecutive mini-batch gradients and leverages its underlying structure making the gradients sequentially predictable. By exploiting the predictability and ideas from optimistic online learning, the proposed algorithm can accelerate the convergence and increase sample efficiency. After establishing a tighter upper bound under some convexity conditions on the regret, we offer a complimentary view of our algorithm which generalizes the offline and stochastic version of nonconvex optimization. In the nonconvex case, we establish a non-asymptotic convergence bound independently of the initialization. We illustrate the practical speedup on several deep learning models via numerical experiments.