Explicit Eigenvalue Regularization Improves Sharpness-Aware Minimization
This work addresses the efficiency and theoretical gaps in SAM for machine learning practitioners, though it is incremental as it builds directly on existing SAM methods.
The paper tackled the limited understanding of Sharpness-Aware Minimization (SAM) by analyzing its training dynamics and proposing Eigen-SAM, which explicitly aligns the perturbation vector with the top Hessian eigenvector to improve sharpness regularization, achieving better generalization in experiments.
Sharpness-Aware Minimization (SAM) has attracted significant attention for its effectiveness in improving generalization across various tasks. However, its underlying principles remain poorly understood. In this work, we analyze SAM's training dynamics using the maximum eigenvalue of the Hessian as a measure of sharpness, and propose a third-order stochastic differential equation (SDE), which reveals that the dynamics are driven by a complex mixture of second- and third-order terms. We show that alignment between the perturbation vector and the top eigenvector is crucial for SAM's effectiveness in regularizing sharpness, but find that this alignment is often inadequate in practice, limiting SAM's efficiency. Building on these insights, we introduce Eigen-SAM, an algorithm that explicitly aims to regularize the top Hessian eigenvalue by aligning the perturbation vector with the leading eigenvector. We validate the effectiveness of our theory and the practical advantages of our proposed approach through comprehensive experiments. Code is available at https://github.com/RitianLuo/EigenSAM.