Large Spikes in Stochastic Gradient Descent: A Large-Deviations View
This work addresses the problem of understanding large spikes in SGD for neural network training, offering a theoretical explanation that is incremental but specific to the NTK regime.
The paper analyzes stochastic gradient descent (SGD) training of a shallow neural network in the neural tangent kernel (NTK) scaling, identifying an explicit criterion that predicts large NTK-flattening spikes with high probability when a function G is positive, and their probability decays like (n/η)^{−ϑ/2} when G is negative, providing a parameter-dependent explanation for spike observations at practical widths.
We analyse SGD training of a shallow, fully connected network in the NTK scaling and provide a quantitative theory of the catapult phase. We identify an explicit criterion separating two behaviours: When an explicit function $G$, depending only on the kernel, learning rate $η$ and data, is positive, SGD produces large NTK-flattening spikes with high probability; when $G<0$, their probability decays like $(n/η)^{-\vartheta/2}$, for an explicitly characterised $\vartheta\in (0,\infty)$. This yields a concrete parameter-dependent explanation for why such spikes may still be observed at practical widths.