Unbiased Supervised Contrastive Learning
It addresses the issue of dataset bias in machine learning, which can lead to models that fail in real-world scenarios, by providing a method to learn robust representations, though it appears incremental as it builds on existing contrastive learning frameworks.
The paper tackles the problem of learning unbiased representations from biased data by proposing a novel supervised contrastive loss (epsilon-SupInfoNCE) and a debiasing regularization loss (FairKL), achieving state-of-the-art performance on biased datasets like CIFAR10, CIFAR100, and ImageNet.
Many datasets are biased, namely they contain easy-to-learn features that are highly correlated with the target class only in the dataset but not in the true underlying distribution of the data. For this reason, learning unbiased models from biased data has become a very relevant research topic in the last years. In this work, we tackle the problem of learning representations that are robust to biases. We first present a margin-based theoretical framework that allows us to clarify why recent contrastive losses (InfoNCE, SupCon, etc.) can fail when dealing with biased data. Based on that, we derive a novel formulation of the supervised contrastive loss (epsilon-SupInfoNCE), providing more accurate control of the minimal distance between positive and negative samples. Furthermore, thanks to our theoretical framework, we also propose FairKL, a new debiasing regularization loss, that works well even with extremely biased data. We validate the proposed losses on standard vision datasets including CIFAR10, CIFAR100, and ImageNet, and we assess the debiasing capability of FairKL with epsilon-SupInfoNCE, reaching state-of-the-art performance on a number of biased datasets, including real instances of biases in the wild.