Catapult Dynamics and Phase Transitions in Quadratic Nets
This work provides theoretical and empirical insights into phase transitions in neural network training, addressing a fundamental problem in machine learning optimization.
The paper proves that the catapult phase, where training loss initially grows before rapidly decreasing, exists in a broad class of models including quadratic and two-layer neural nets, and shows empirically that ReLU nets develop sparser activations with higher learning rates.
Neural networks trained with gradient descent can undergo non-trivial phase transitions as a function of the learning rate. In \cite{lewkowycz2020large} it was discovered that wide neural nets can exhibit a catapult phase for super-critical learning rates, where the training loss grows exponentially quickly at early times before rapidly decreasing to a small value. During this phase the top eigenvalue of the neural tangent kernel (NTK) also undergoes significant evolution. In this work, we will prove that the catapult phase exists in a large class of models, including quadratic models and two-layer, homogenous neural nets. To do this, we show that for a certain range of learning rates the weight norm decreases whenever the loss becomes large. We also empirically study learning rates beyond this theoretically derived range and show that the activation map of ReLU nets trained with super-critical learning rates becomes increasingly sparse as we increase the learning rate.