There is a Singularity in the Loss Landscape
This addresses a fundamental problem in understanding neural network training dynamics for researchers and practitioners, though it is incremental in explaining existing observations.
The paper identifies a singularity in the neural network loss landscape where gradient magnitude becomes unbounded as dataset size increases, explaining phenomena like training on the edge of stability and gradient concentration in a top subspace.
Despite the widespread adoption of neural networks, their training dynamics remain poorly understood. We show experimentally that as the size of the dataset increases, a point forms where the magnitude of the gradient of the loss becomes unbounded. Gradient descent rapidly brings the network close to this singularity in parameter space, and further training takes place near it. This singularity explains a variety of phenomena recently observed in the Hessian of neural network loss functions, such as training on the edge of stability and the concentration of the gradient in a top subspace. Once the network approaches the singularity, the top subspace contributes little to learning, even though it constitutes the majority of the gradient.