GA-SAM: Gradient-Strength based Adaptive Sharpness-Aware Minimization for Improved Generalization
This work addresses a problem for natural language processing researchers by providing an incremental improvement to SAM for tasks with drastic gradient changes.
The authors tackled the difficulty of applying Sharpness-Aware Minimization (SAM) to natural language tasks with models like RNNs by proposing GA-SAM, a gradient-strength based adaptive algorithm that improves generalization, as validated on various language benchmarks.
Recently, Sharpness-Aware Minimization (SAM) algorithm has shown state-of-the-art generalization abilities in vision tasks. It demonstrates that flat minima tend to imply better generalization abilities. However, it has some difficulty implying SAM to some natural language tasks, especially to models with drastic gradient changes, such as RNNs. In this work, we analyze the relation between the flatness of the local minimum and its generalization ability from a novel and straightforward theoretical perspective. We propose that the shift of the training and test distributions can be equivalently seen as a virtual parameter corruption or perturbation, which can explain why flat minima that are robust against parameter corruptions or perturbations have better generalization performances. On its basis, we propose a Gradient-Strength based Adaptive Sharpness-Aware Minimization (GA-SAM) algorithm to help to learn algorithms find flat minima that generalize better. Results in various language benchmarks validate the effectiveness of the proposed GA-SAM algorithm on natural language tasks.