On Convergence of Adam for Stochastic Optimization under Relaxed Assumptions
This provides theoretical guarantees for Adam in challenging stochastic optimization settings, which is incremental but important for practitioners in machine learning.
The authors tackled the limited theoretical understanding of the vanilla Adam optimizer in non-convex smooth scenarios with unbounded gradients and affine variance noise, showing that Adam achieves a convergence rate of O(poly(log T)/√T) in high probability and is free of tuning step-sizes with problem parameters.
The Adaptive Momentum Estimation (Adam) algorithm is highly effective in training various deep learning tasks. Despite this, there's limited theoretical understanding for Adam, especially when focusing on its vanilla form in non-convex smooth scenarios with potential unbounded gradients and affine variance noise. In this paper, we study vanilla Adam under these challenging conditions. We introduce a comprehensive noise model which governs affine variance noise, bounded noise and sub-Gaussian noise. We show that Adam can find a stationary point with a $\mathcal{O}(\text{poly}(\log T)/\sqrt{T})$ rate in high probability under this general noise model where $T$ denotes total number iterations, matching the lower rate of stochastic first-order algorithms up to logarithm factors. More importantly, we reveal that Adam is free of tuning step-sizes with any problem-parameters, yielding a better adaptation property than the Stochastic Gradient Descent under the same conditions. We also provide a probabilistic convergence result for Adam under a generalized smooth condition which allows unbounded smoothness parameters and has been illustrated empirically to more accurately capture the smooth property of many practical objective functions.