A note on diffusion limits for stochastic gradient descent
This provides a rigorous foundation for understanding noise in SGD, which is incremental but clarifies existing practices in machine learning theory.
The paper tackles the theoretical justification for approximating stochastic gradient descent (SGD) with a stochastic differential equation using Gaussian noise, showing how this Gaussianity arises naturally in the diffusion limits.
In the machine learning literature stochastic gradient descent has recently been widely discussed for its purported implicit regularization properties. Much of the theory, that attempts to clarify the role of noise in stochastic gradient algorithms, has widely approximated stochastic gradient descent by a stochastic differential equation with Gaussian noise. We provide a novel rigorous theoretical justification for this practice that showcases how the Gaussianity of the noise arises naturally.