SAdam: A Variant of Adam for Strongly Convex Functions
This work addresses a theoretical bottleneck in optimization algorithms for machine learning practitioners, offering incremental improvements for strongly convex scenarios.
The paper tackles the problem of improving Adam's performance for strongly convex functions by developing SAdam, a variant that achieves a data-dependent O(log T) regret bound, compared to the O(sqrt(T)) bound of standard Adam, with empirical results demonstrating effectiveness.
The Adam algorithm has become extremely popular for large-scale machine learning. Under convexity condition, it has been proved to enjoy a data-dependant $O(\sqrt{T})$ regret bound where $T$ is the time horizon. However, whether strong convexity can be utilized to further improve the performance remains an open problem. In this paper, we give an affirmative answer by developing a variant of Adam (referred to as SAdam) which achieves a data-dependant $O(\log T)$ regret bound for strongly convex functions. The essential idea is to maintain a faster decaying yet under controlled step size for exploiting strong convexity. In addition, under a special configuration of hyperparameters, our SAdam reduces to SC-RMSprop, a recently proposed variant of RMSprop for strongly convex functions, for which we provide the first data-dependent logarithmic regret bound. Empirical results on optimizing strongly convex functions and training deep networks demonstrate the effectiveness of our method.