Increasing Both Batch Size and Learning Rate Accelerates Stochastic Gradient Descent
This addresses the optimization challenge for machine learning practitioners by providing theoretical and numerical evidence for more efficient training schedules, though it is incremental as it builds on existing SGD methods.
The paper tackles the problem of optimizing batch size and learning rate schedulers for mini-batch stochastic gradient descent (SGD) in deep neural networks, showing that increasing both batch size and learning rate (or with warm-up) accelerates SGD and minimizes the full gradient norm faster than traditional schedulers.
The performance of mini-batch stochastic gradient descent (SGD) strongly depends on setting the batch size and learning rate to minimize the empirical loss in training the deep neural network. In this paper, we present theoretical analyses of mini-batch SGD with four schedulers: (i) constant batch size and decaying learning rate scheduler, (ii) increasing batch size and decaying learning rate scheduler, (iii) increasing batch size and increasing learning rate scheduler, and (iv) increasing batch size and warm-up decaying learning rate scheduler. We show that mini-batch SGD using scheduler (i) does not always minimize the expectation of the full gradient norm of the empirical loss, whereas it does using any of schedulers (ii), (iii), and (iv). Furthermore, schedulers (iii) and (iv) accelerate mini-batch SGD. The paper also provides numerical results of supporting analyses showing that using scheduler (iii) or (iv) minimizes the full gradient norm of the empirical loss faster than using scheduler (i) or (ii).