Convergence Analysis of a Momentum Algorithm with Adaptive Step Size for Non Convex Optimization
This work addresses convergence problems in popular optimization algorithms like ADAM for machine learning practitioners, but it is incremental as it builds on existing variants to provide theoretical guarantees.
The paper tackles the convergence issues of the ADAM algorithm in non-convex optimization by proposing a bounded adaptive step size based on the Lipschitz constant, showing novel first-order convergence rates in deterministic and stochastic settings, and establishing rates for function values using the Kurdyka-Lojasiewicz property.
Although ADAM is a very popular algorithm for optimizing the weights of neural networks, it has been recently shown that it can diverge even in simple convex optimization examples. Several variants of ADAM have been proposed to circumvent this convergence issue. In this work, we study the ADAM algorithm for smooth nonconvex optimization under a boundedness assumption on the adaptive learning rate. The bound on the adaptive step size depends on the Lipschitz constant of the gradient of the objective function and provides safe theoretical adaptive step sizes. Under this boundedness assumption, we show a novel first order convergence rate result in both deterministic and stochastic contexts. Furthermore, we establish convergence rates of the function value sequence using the Kurdyka-Lojasiewicz property.