Improving Resistance to Noisy Label Fitting by Reweighting Gradient in SAM
This work addresses noisy label challenges in classification tasks, offering an incremental improvement over SAM for better generalization in realistic training settings.
The paper tackled the problem of noisy labels in machine learning by proposing SANER, a variant of Sharpness-Aware Minimization (SAM), which improved robustness against noisy fitting and achieved up to an 8% increase in accuracy on CIFAR-100 with 50% label noise.
Noisy labels pose a substantial challenge in machine learning, often resulting in overfitting and poor generalization. Sharpness-Aware Minimization (SAM), as demonstrated in Foret et al. (2021), improves generalization over traditional Stochastic Gradient Descent (SGD) in classification tasks with noisy labels by implicitly slowing noisy learning. While SAM's ability to generalize in noisy environments has been studied in several simplified settings, its full potential in more realistic training settings remains underexplored. In this work, we analyze SAM's behavior at each iteration, identifying specific components of the gradient vector that contribute significantly to its robustness against noisy labels. Based on these insights, we propose SANER (Sharpness-Aware Noise-Explicit Reweighting), an effective variant that enhances SAM's ability to manage noisy fitting rate. Our experiments on CIFAR-10, CIFAR-100, and Mini-WebVision demonstrate that SANER consistently outperforms SAM, achieving up to an 8% increase on CIFAR-100 with 50% label noise.