Sharpness-Aware Minimization with Adaptive Regularization for Training Deep Neural Networks
This work addresses the need for more flexible regularization in SAM to enhance model generalization, though it is incremental as it builds directly on existing SAM methods.
The authors tackled the fixed hyperparameter limitation in Sharpness-Aware Minimization (SAM) by proposing SAMAR, which uses adaptive regularization to dynamically update the parameter, resulting in improved accuracy and generalization on CIFAR-10 and CIFAR-100 image recognition tasks.
Sharpness-Aware Minimization (SAM) has proven highly effective in improving model generalization in machine learning tasks. However, SAM employs a fixed hyperparameter associated with the regularization to characterize the sharpness of the model. Despite its success, research on adaptive regularization methods based on SAM remains scarce. In this paper, we propose the SAM with Adaptive Regularization (SAMAR), which introduces a flexible sharpness ratio rule to update the regularization parameter dynamically. We provide theoretical proof of the convergence of SAMAR for functions satisfying the Lipschitz continuity. Additionally, experiments on image recognition tasks using CIFAR-10 and CIFAR-100 demonstrate that SAMAR enhances accuracy and model generalization.