The instabilities of large learning rate training: a loss landscape view
This addresses training stability issues for practitioners using large learning rates, though it appears incremental as it builds on existing loss landscape studies.
The paper investigates the instabilities in neural network training with large learning rates by analyzing the Hessian matrix, revealing phenomena like landscape flattening and shift that are linked to these instabilities.
Modern neural networks are undeniably successful. Numerous works study how the curvature of loss landscapes can affect the quality of solutions. In this work we study the loss landscape by considering the Hessian matrix during network training with large learning rates - an attractive regime that is (in)famously unstable. We characterise the instabilities of gradient descent, and we observe the striking phenomena of \textit{landscape flattening} and \textit{landscape shift}, both of which are intimately connected to the instabilities of training.