Why Does Multi-Epoch Training Help?
This work addresses a fundamental theoretical problem in machine learning optimization, providing insights for researchers and practitioners using SGD in training deep neural networks, though it is incremental as it builds on existing conditions.
The paper tackles the theoretical gap between empirical observations and standard risk bounds for multi-epoch SGD, showing that under the Polyak-Lojasiewicz condition, multi-pass SGD achieves a faster convergence rate in excess risk bounds compared to one-pass SGD for smooth non-convex least squared loss problems.
Stochastic gradient descent (SGD) has become the most attractive optimization method in training large-scale deep neural networks due to its simplicity, low computational cost in each updating step, and good performance. Standard excess risk bounds show that SGD only needs to take one pass over the training data and more passes could not help to improve the performance. Empirically, it has been observed that SGD taking more than one pass over the training data (multi-pass SGD) has much better excess risk bound performance than the SGD only taking one pass over the training data (one-pass SGD). However, it is not very clear that how to explain this phenomenon in theory. In this paper, we provide some theoretical evidences for explaining why multiple passes over the training data can help improve performance under certain circumstance. Specifically, we consider smooth risk minimization problems whose objective function is non-convex least squared loss. Under Polyak-Lojasiewicz (PL) condition, we establish faster convergence rate of excess risk bound for multi-pass SGD than that for one-pass SGD.