An Empirical Study of Large-Batch Stochastic Gradient Descent with Structured Covariance Noise
This addresses a key challenge in deep learning for practitioners by enabling faster training with large batches without sacrificing model accuracy, though it appears incremental as it builds on existing noise-based methods.
The paper tackles the problem of improving generalization in large-batch stochastic gradient descent without harming optimization convergence by adding structured covariance noise to gradients, showing that this method enhances generalization performance while maintaining desirable optimization and training duration in deep learning models.
The choice of batch-size in a stochastic optimization algorithm plays a substantial role for both optimization and generalization. Increasing the batch-size used typically improves optimization but degrades generalization. To address the problem of improving generalization while maintaining optimal convergence in large-batch training, we propose to add covariance noise to the gradients. We demonstrate that the learning performance of our method is more accurately captured by the structure of the covariance matrix of the noise rather than by the variance of gradients. Moreover, over the convex-quadratic, we prove in theory that it can be characterized by the Frobenius norm of the noise matrix. Our empirical studies with standard deep learning model-architectures and datasets shows that our method not only improves generalization performance in large-batch training, but furthermore, does so in a way where the optimization performance remains desirable and the training duration is not elongated.