Convergence of Sharpness-Aware Minimization Algorithms using Increasing Batch Size and Decaying Learning Rate
This work addresses the challenge of finding flat minima for better generalization in deep learning, but it is incremental as it builds on existing SAM and GSAM methods by incorporating known optimization techniques.
The paper tackles the problem of improving generalization in deep neural networks by theoretically proving the convergence of the GSAM algorithm with increasing batch sizes or decaying learning rates, and numerically shows that these strategies find flatter local minima, with specific comparisons indicating improved flatness over constant settings.
The sharpness-aware minimization (SAM) algorithm and its variants, including gap guided SAM (GSAM), have been successful at improving the generalization capability of deep neural network models by finding flat local minima of the empirical loss in training. Meanwhile, it has been shown theoretically and practically that increasing the batch size or decaying the learning rate avoids sharp local minima of the empirical loss. In this paper, we consider the GSAM algorithm with increasing batch sizes or decaying learning rates, such as cosine annealing or linear learning rate, and theoretically show its convergence. Moreover, we numerically compare SAM (GSAM) with and without an increasing batch size and conclude that using an increasing batch size or decaying learning rate finds flatter local minima than using a constant batch size and learning rate.