Divergence of the ADAM algorithm with fixed-stepsize: a (very) simple example

This work identifies a fundamental flaw in ADAM's convergence guarantees for practitioners in optimization and machine learning, showing it is incremental by providing a counterexample to existing assumptions.

The authors constructed a simple unidimensional function with Lipschitz continuous gradient to demonstrate that the ADAM algorithm with constant stepsize diverges when minimizing it from the origin without gradient noise, regardless of parameter choices.