Emergence of heavy tails in homogenized stochastic gradient descent
This work contributes to understanding the heavy-tailed behavior in SGD, which is relevant for researchers studying generalization and optimization in neural networks.
The paper analyzes a continuous diffusion approximation of stochastic gradient descent (SGD) called homogenized SGD, showing it exhibits heavy-tailed parameter distributions and providing explicit bounds on the tail-index, validated through numerical experiments.
It has repeatedly been observed that loss minimization by stochastic gradient descent (SGD) leads to heavy-tailed distributions of neural network parameters. Here, we analyze a continuous diffusion approximation of SGD, called homogenized stochastic gradient descent, show that it behaves asymptotically heavy-tailed, and give explicit upper and lower bounds on its tail-index. We validate these bounds in numerical experiments and show that they are typically close approximations to the empirical tail-index of SGD iterates. In addition, their explicit form enables us to quantify the interplay between optimization parameters and the tail-index. Doing so, we contribute to the ongoing discussion on links between heavy tails and the generalization performance of neural networks as well as the ability of SGD to avoid suboptimal local minima.