Flatter, faster: scaling momentum for optimal speedup of SGD
This provides a principled scaling rule for momentum in SGD, addressing a common bottleneck in optimizing neural networks, though it is incremental as it builds on existing momentum methods.
The authors tackled the problem of selecting the momentum hyperparameter in SGD to accelerate training without sacrificing generalization, finding that scaling momentum with the learning rate to the power of 2/3 maximally speeds up training, as confirmed on synthetic and realistic datasets like CIFAR10 and FashionMNIST.
Commonly used optimization algorithms often show a trade-off between good generalization and fast training times. For instance, stochastic gradient descent (SGD) tends to have good generalization; however, adaptive gradient methods have superior training times. Momentum can help accelerate training with SGD, but so far there has been no principled way to select the momentum hyperparameter. Here we study training dynamics arising from the interplay between SGD with label noise and momentum in the training of overparametrized neural networks. We find that scaling the momentum hyperparameter $1-β$ with the learning rate to the power of $2/3$ maximally accelerates training, without sacrificing generalization. To analytically derive this result we develop an architecture-independent framework, where the main assumption is the existence of a degenerate manifold of global minimizers, as is natural in overparametrized models. Training dynamics display the emergence of two characteristic timescales that are well-separated for generic values of the hyperparameters. The maximum acceleration of training is reached when these two timescales meet, which in turn determines the scaling limit we propose. We confirm our scaling rule for synthetic regression problems (matrix sensing and teacher-student paradigm) and classification for realistic datasets (ResNet-18 on CIFAR10, 6-layer MLP on FashionMNIST), suggesting the robustness of our scaling rule to variations in architectures and datasets.