When Do Flat Minima Optimizers Work?
This work addresses the problem of selecting effective optimizers for deep learning practitioners by providing empirical insights, though it is incremental as it benchmarks existing methods without introducing new ones.
The paper tackled the lack of systematic benchmarking of flat-minima optimizers like SWA and SAM by comparing their loss surfaces and evaluating them across computer vision, NLP, and graph learning tasks, revealing surprising findings to guide researchers and practitioners.
Recently, flat-minima optimizers, which seek to find parameters in low-loss neighborhoods, have been shown to improve a neural network's generalization performance over stochastic and adaptive gradient-based optimizers. Two methods have received significant attention due to their scalability: 1. Stochastic Weight Averaging (SWA), and 2. Sharpness-Aware Minimization (SAM). However, there has been limited investigation into their properties and no systematic benchmarking of them across different domains. We fill this gap here by comparing the loss surfaces of the models trained with each method and through broad benchmarking across computer vision, natural language processing, and graph representation learning tasks. We discover several surprising findings from these results, which we hope will help researchers further improve deep learning optimizers, and practitioners identify the right optimizer for their problem.