LightSAM: Parameter-Agnostic Sharpness-Aware Minimization
This addresses the parameter-tuning burden for practitioners using SAM in deep learning, though it is incremental as it builds directly on SAM.
The paper tackles the sensitivity of Sharpness-Aware Minimization (SAM) to hyper-parameters like perturbation radius and learning rate by proposing LightSAM, which adaptively sets these parameters using adaptive optimizers, achieving parameter-agnostic convergence and validating effectiveness in experiments.
Sharpness-Aware Minimization (SAM) optimizer enhances the generalization ability of the machine learning model by exploring the flat minima landscape through weight perturbations. Despite its empirical success, SAM introduces an additional hyper-parameter, the perturbation radius, which causes the sensitivity of SAM to it. Moreover, it has been proved that the perturbation radius and learning rate of SAM are constrained by problem-dependent parameters to guarantee convergence. These limitations indicate the requirement of parameter-tuning in practical applications. In this paper, we propose the algorithm LightSAM which sets the perturbation radius and learning rate of SAM adaptively, thus extending the application scope of SAM. LightSAM employs three popular adaptive optimizers, including AdaGrad-Norm, AdaGrad and Adam, to replace the SGD optimizer for weight perturbation and model updating, reducing sensitivity to parameters. Theoretical results show that under weak assumptions, LightSAM could converge ideally with any choices of perturbation radius and learning rate, thus achieving parameter-agnostic. We conduct preliminary experiments on several deep learning tasks, which together with the theoretical findings validate the the effectiveness of LightSAM.