Zeroth-Order Sharpness-Aware Learning with Exponential Tilting
This work provides a gradient-free and memory-efficient alternative to SAM variants for improving generalization in machine learning, though it is incremental as it builds on existing SAM and zeroth-order methods.
The paper tackles the challenge of connecting zeroth-order optimization with sharpness-aware minimization (SAM) by introducing an exponential tilting objective that smoothly transitions between average- and max-loss formulations, resulting in better generalization on tasks like classification, QA, and language generation compared to vanilla zeroth-order baselines.
Classic zeroth-order optimization approaches typically optimize for a smoothed version of the original function, i.e., the expected objective under randomly perturbed model parameters. This can be interpreted as encouraging the loss values in the perturbation set to be small on average. Popular sharpness-aware minimization (SAM) objectives, however, typically focus on the largest loss within the neighborhood to arrive at flat minima more effectively. In this work, we connect zeroth-order optimization (and its corresponding objectives) with SAM approaches explicitly, through an exponential tilting objective that provides a smooth transition between the average- and the max-loss formulations. We explore new zeroth-order algorithms to solve a soft SAM objective parameterized by a tilting parameter $t$. We provide precise characterizations of the sharpness notions of the tilted SAM framework. Practically, our approach can be used as a gradient-free and memory-efficient alternative to SAM variants, and it achieves better generalization compared to vanilla zeroth-order baselines on a wide range of downstream tasks, including classification, multiple choice QA, and language generation.