Preconditioned Sharpness-Aware Minimization: Unifying Analysis and a Novel Learning Algorithm
This work addresses the problem of improving generalizability in deep neural networks for researchers and practitioners, offering a novel algorithm that builds incrementally on existing SAM methods.
The paper tackles the lack of a unifying approach for sharpness-aware minimization (SAM) variants by introducing preconditioning to unify them and provide convergence analysis, then proposes infoSAM to address adversarial model degradation by adjusting gradients based on noise estimates, achieving superior performance in benchmarks.
Targeting solutions over `flat' regions of the loss landscape, sharpness-aware minimization (SAM) has emerged as a powerful tool to improve generalizability of deep neural network based learning. While several SAM variants have been developed to this end, a unifying approach that also guides principled algorithm design has been elusive. This contribution leverages preconditioning (pre) to unify SAM variants and provide not only unifying convergence analysis, but also valuable insights. Building upon preSAM, a novel algorithm termed infoSAM is introduced to address the so-called adversarial model degradation issue in SAM by adjusting gradients depending on noise estimates. Extensive numerical tests demonstrate the superiority of infoSAM across various benchmarks.