Improved generalization by noise enhancement
This addresses a scalability issue in deep learning by enabling stable, parallelizable training with improved generalization, though it is incremental as it builds on known SGD noise relationships.
The paper tackles the problem of controlling SGD noise to improve generalization without altering learning rate or batch size, proposing a noise enhancement method that achieves better generalization on real datasets, with large-batch training using this method outperforming small-batch training.
Recent studies have demonstrated that noise in stochastic gradient descent (SGD) is closely related to generalization: A larger SGD noise, if not too large, results in better generalization. Since the covariance of the SGD noise is proportional to $η^2/B$, where $η$ is the learning rate and $B$ is the minibatch size of SGD, the SGD noise has so far been controlled by changing $η$ and/or $B$. However, too large $η$ results in instability in the training dynamics and a small $B$ prevents scalable parallel computation. It is thus desirable to develop a method of controlling the SGD noise without changing $η$ and $B$. In this paper, we propose a method that achieves this goal using ``noise enhancement'', which is easily implemented in practice. We expound the underlying theoretical idea and demonstrate that the noise enhancement actually improves generalization for real datasets. It turns out that large-batch training with the noise enhancement even shows better generalization compared with small-batch training.