Agnostic Sharpness-Aware Minimization
This work addresses generalization challenges in deep learning for practitioners dealing with noisy or limited data, representing an incremental hybrid approach.
The paper tackled the problem of improving deep neural network generalization by combining sharpness-aware minimization (SAM) and model-agnostic meta-learning (MAML) into Agnostic-SAM, which optimizes for flatter minima robust to perturbations and distribution shifts, resulting in significant generalization improvements across datasets under noisy labels or data limitations.
Sharpness-aware minimization (SAM) has been instrumental in improving deep neural network training by minimizing both the training loss and the sharpness of the loss landscape, leading the model into flatter minima that are associated with better generalization properties. In another aspect, Model-Agnostic Meta-Learning (MAML) is a framework designed to improve the adaptability of models. MAML optimizes a set of meta-models that are specifically tailored for quick adaptation to multiple tasks with minimal fine-tuning steps and can generalize well with limited data. In this work, we explore the connection between SAM and MAML in enhancing model generalization. We introduce Agnostic-SAM, a novel approach that combines the principles of both SAM and MAML. Agnostic-SAM adapts the core idea of SAM by optimizing the model toward wider local minima using training data, while concurrently maintaining low loss values on validation data. By doing so, it seeks flatter minima that are not only robust to small perturbations but also less vulnerable to data distributional shift problems. Our experimental results demonstrate that Agnostic-SAM significantly improves generalization over baselines across a range of datasets and under challenging conditions such as noisy labels or data limitation.