On the Convergence of AdaBound and its Connection to SGD
This work addresses convergence issues in adaptive optimizers like AdaBound, which is important for researchers and practitioners in deep learning, though it is incremental as it builds on known limitations.
The paper identifies a stochastic convex problem where AdaBound can take arbitrarily long to converge, contrary to its theoretical guarantees, and shows that momentum SGD can match AdaBound's performance on CIFAR with fewer hyperparameters and lower costs.
Adaptive gradient methods such as Adam have gained extreme popularity due to their success in training complex neural networks and less sensitivity to hyperparameter tuning compared to SGD. However, it has been recently shown that Adam can fail to converge and might cause poor generalization -- this lead to the design of new, sophisticated adaptive methods which attempt to generalize well while being theoretically reliable. In this technical report we focus on AdaBound, a promising, recently proposed optimizer. We present a stochastic convex problem for which AdaBound can provably take arbitrarily long to converge in terms of a factor which is not accounted for in the convergence rate guarantee of Luo et al. (2019). We present a new $O(\sqrt T)$ regret guarantee under different assumptions on the bound functions, and provide empirical results on CIFAR suggesting that a specific form of momentum SGD can match AdaBound's performance while having less hyperparameters and lower computational costs.