SGD with a Constant Large Learning Rate Can Converge to Local Maxima
This work highlights potential pitfalls of SGD in deep learning for practitioners, though it is incremental as it focuses on constructed worst-case scenarios rather than typical training regimes.
The paper constructs worst-case optimization problems showing that stochastic gradient descent (SGD) with a constant large learning rate can converge to local maxima, escape saddle points arbitrarily slowly, prefer sharp minima over flat ones, and cause AMSGrad to converge to local maxima, with examples in neural network-like settings.
Previous works on stochastic gradient descent (SGD) often focus on its success. In this work, we construct worst-case optimization problems illustrating that, when not in the regimes that the previous works often assume, SGD can exhibit many strange and potentially undesirable behaviors. Specifically, we construct landscapes and data distributions such that (1) SGD converges to local maxima, (2) SGD escapes saddle points arbitrarily slowly, (3) SGD prefers sharp minima over flat ones, and (4) AMSGrad converges to local maxima. We also realize results in a minimal neural network-like example. Our results highlight the importance of simultaneously analyzing the minibatch sampling, discrete-time updates rules, and realistic landscapes to understand the role of SGD in deep learning.