Towards Theoretical Understanding of Large Batch Training in Stochastic Gradient Descent
This work addresses the theoretical understanding of optimization dynamics in deep learning, providing insights for practitioners on batch size and learning rate tuning, though it is incremental in building on prior hypotheses.
The paper theoretically investigates the hypothesis that large batch training in stochastic gradient descent (SGD) leads to sharp minimizers, showing that SGD tends to converge to flatter minima asymptotically regardless of batch size, but with trade-offs in convergence speed and generalization performance.
Stochastic gradient descent (SGD) is almost ubiquitously used for training non-convex optimization tasks. Recently, a hypothesis proposed by Keskar et al. [2017] that large batch methods tend to converge to sharp minimizers has received increasing attention. We theoretically justify this hypothesis by providing new properties of SGD in both finite-time and asymptotic regimes. In particular, we give an explicit escaping time of SGD from a local minimum in the finite-time regime and prove that SGD tends to converge to flatter minima in the asymptotic regime (although may take exponential time to converge) regardless of the batch size. We also find that SGD with a larger ratio of learning rate to batch size tends to converge to a flat minimum faster, however, its generalization performance could be worse than the SGD with a smaller ratio of learning rate to batch size. We include numerical experiments to corroborate these theoretical findings.