UAdam: Unified Adam-Type Algorithmic Framework for Non-Convex Stochastic Optimization
This provides theoretical guarantees for Adam-type algorithms, which are widely used in deep learning, though it is incremental as it builds on existing methods.
The authors tackled the problem of understanding convergence for Adam-type optimization algorithms by introducing UAdam, a unified framework that includes Adam and its variants, and proved it converges to stationary points with a rate of O(1/T) in non-convex stochastic settings.
Adam-type algorithms have become a preferred choice for optimisation in the deep learning setting, however, despite success, their convergence is still not well understood. To this end, we introduce a unified framework for Adam-type algorithms (called UAdam). This is equipped with a general form of the second-order moment, which makes it possible to include Adam and its variants as special cases, such as NAdam, AMSGrad, AdaBound, AdaFom, and Adan. This is supported by a rigorous convergence analysis of UAdam in the non-convex stochastic setting, showing that UAdam converges to the neighborhood of stationary points with the rate of $\mathcal{O}(1/T)$. Furthermore, the size of neighborhood decreases as $β$ increases. Importantly, our analysis only requires the first-order momentum factor to be close enough to 1, without any restrictions on the second-order momentum factor. Theoretical results also show that vanilla Adam can converge by selecting appropriate hyperparameters, which provides a theoretical guarantee for the analysis, applications, and further developments of the whole class of Adam-type algorithms.