ODE approximation for the Adam algorithm: General and overparametrized setting
This provides theoretical insights into Adam's convergence properties, which is incremental for deep learning practitioners and researchers.
The authors tackled the problem of understanding the convergence behavior of the Adam optimizer by developing an ODE-based method to analyze it in fast-slow scaling regimes, showing that in general settings, Adam converges to zeros of a vector field rather than local minima, but in overparametrized cases, it can find global minima if it enters their neighborhood infinitely often.
The Adam optimizer is currently presumably the most popular optimization method in deep learning. In this article we develop an ODE based method to study the Adam optimizer in a fast-slow scaling regime. For fixed momentum parameters and vanishing step-sizes, we show that the Adam algorithm is an asymptotic pseudo-trajectory of the flow of a particular vector field, which is referred to as the Adam vector field. Leveraging properties of asymptotic pseudo-trajectories, we establish convergence results for the Adam algorithm. In particular, in a very general setting we show that if the Adam algorithm converges, then the limit must be a zero of the Adam vector field, rather than a local minimizer or critical point of the objective function. In contrast, in the overparametrized empirical risk minimization setting, the Adam algorithm is able to locally find the set of minima. Specifically, we show that in a neighborhood of the global minima, the objective function serves as a Lyapunov function for the flow induced by the Adam vector field. As a consequence, if the Adam algorithm enters a neighborhood of the global minima infinitely often, it converges to the set of global minima.