SAM operates far from home: eigenvalue regularization as a dynamical phenomenon
This provides a deeper understanding of SAM's regularization effects for machine learning practitioners, though it is incremental as it builds on existing SAM and edge of stability analyses.
The paper reveals that the Sharpness Aware Minimization (SAM) algorithm dynamically regularizes eigenvalues throughout the learning trajectory, not just near minima, and shows this relates to the edge of stability phenomenon, with theory predicting the largest eigenvalue based on learning rate and SAM radius parameters.
The Sharpness Aware Minimization (SAM) optimization algorithm has been shown to control large eigenvalues of the loss Hessian and provide generalization benefits in a variety of settings. The original motivation for SAM was a modified loss function which penalized sharp minima; subsequent analyses have also focused on the behavior near minima. However, our work reveals that SAM provides a strong regularization of the eigenvalues throughout the learning trajectory. We show that in a simplified setting, SAM dynamically induces a stabilization related to the edge of stability (EOS) phenomenon observed in large learning rate gradient descent. Our theory predicts the largest eigenvalue as a function of the learning rate and SAM radius parameters. Finally, we show that practical models can also exhibit this EOS stabilization, and that understanding SAM must account for these dynamics far away from any minima.