Second-order regression models exhibit progressive sharpening to the edge of stability
This work addresses the understanding of training dynamics in non-linear models for machine learning researchers, showing that key behaviors are more general, though it is incremental as it extends known concepts to simpler models.
The paper investigates whether progressive sharpening and edge-of-stability phenomena, observed in neural networks with large gradient descent step sizes, occur in simpler second-order regression models. It proves that these models exhibit similar behavior in two dimensions and suggests it generalizes to higher dimensions, indicating these phenomena are not unique to neural networks.
Recent studies of gradient descent with large step sizes have shown that there is often a regime with an initial increase in the largest eigenvalue of the loss Hessian (progressive sharpening), followed by a stabilization of the eigenvalue near the maximum value which allows convergence (edge of stability). These phenomena are intrinsically non-linear and do not happen for models in the constant Neural Tangent Kernel (NTK) regime, for which the predictive function is approximately linear in the parameters. As such, we consider the next simplest class of predictive models, namely those that are quadratic in the parameters, which we call second-order regression models. For quadratic objectives in two dimensions, we prove that this second-order regression model exhibits progressive sharpening of the NTK eigenvalue towards a value that differs slightly from the edge of stability, which we explicitly compute. In higher dimensions, the model generically shows similar behavior, even without the specific structure of a neural network, suggesting that progressive sharpening and edge-of-stability behavior aren't unique features of neural networks, and could be a more general property of discrete learning algorithms in high-dimensional non-linear models.