Non-Gaussianity of Stochastic Gradient Noise
This addresses a fundamental question in machine learning about optimization and generalization, but the findings are incremental as they confirm Gaussianity under specific conditions rather than revealing new mechanisms.
The paper investigates the distribution of Stochastic Gradient Noise (SGN) in neural network training to understand why SGD generalizes better than GD, finding that for batch sizes of 256 and above, the distribution is Gaussian in early training phases across various datasets and architectures.
What enables Stochastic Gradient Descent (SGD) to achieve better generalization than Gradient Descent (GD) in Neural Network training? This question has attracted much attention. In this paper, we study the distribution of the Stochastic Gradient Noise (SGN) vectors during the training. We observe that for batch sizes 256 and above, the distribution is best described as Gaussian at-least in the early phases of training. This holds across data-sets, architectures, and other choices.