An Inclusive Theoretical Framework of Robust Supervised Contrastive Loss against Label Noise
This work addresses a critical challenge in machine learning with broad implications for real-world applications, though it is incremental as it builds on existing supervised contrastive learning approaches.
The authors tackled the problem of learning from noisy labels by proposing a unified theoretical framework for robust supervised contrastive losses, deriving a general condition to verify robustness and developing SymNCE, which outperforms existing methods on benchmark datasets.
Learning from noisy labels is a critical challenge in machine learning, with vast implications for numerous real-world scenarios. While supervised contrastive learning has recently emerged as a powerful tool for navigating label noise, many existing solutions remain heuristic, often devoid of a systematic theoretical foundation for crafting robust supervised contrastive losses. To address the gap, in this paper, we propose a unified theoretical framework for robust losses under the pairwise contrastive paradigm. In particular, we for the first time derive a general robust condition for arbitrary contrastive losses, which serves as a criterion to verify the theoretical robustness of a supervised contrastive loss against label noise. The theory indicates that the popular InfoNCE loss is in fact non-robust, and accordingly inspires us to develop a robust version of InfoNCE, termed Symmetric InfoNCE (SymNCE). Moreover, we highlight that our theory is an inclusive framework that provides explanations to prior robust techniques such as nearest-neighbor (NN) sample selection and robust contrastive loss. Validation experiments on benchmark datasets demonstrate the superiority of SymNCE against label noise.