Does Adam optimizer keep close to the optimal point?
This addresses a fundamental convergence problem in optimization for machine learning practitioners, though it is incremental as it builds on known counterexamples.
The paper identifies a limitation of the Adam optimizer where it fails to converge to the optimal point in certain convex regions, and proposes a new algorithm that overcomes this issue to ensure convergence.
The adaptive optimizer for training neural networks has continually evolved to overcome the limitations of the previously proposed adaptive methods. Recent studies have found the rare counterexamples that Adam cannot converge to the optimal point. Those counterexamples reveal the distortion of Adam due to a small second momentum from a small gradient. Unlike previous studies, we show Adam cannot keep closer to the optimal point for not only the counterexamples but also a general convex region when the effective learning rate exceeds the certain bound. Subsequently, we propose an algorithm that overcomes Adam's limitation and ensures that it can reach and stay at the optimal point region.