Ranking Info Noise Contrastive Estimation: Boosting Contrastive Learning via Ranked Positives
This addresses the problem of learning embeddings when similarity rankings are available or definitions are ambiguous, primarily for researchers in contrastive learning, though it is incremental as it builds on existing InfoNCE methods.
The paper tackles the limitation of standard InfoNCE loss by introducing RINCE, which uses ranked positive samples to improve embedding quality, resulting in higher classification accuracy, retrieval rates, and better out-of-distribution detection compared to InfoNCE.
This paper introduces Ranking Info Noise Contrastive Estimation (RINCE), a new member in the family of InfoNCE losses that preserves a ranked ordering of positive samples. In contrast to the standard InfoNCE loss, which requires a strict binary separation of the training pairs into similar and dissimilar samples, RINCE can exploit information about a similarity ranking for learning a corresponding embedding space. We show that the proposed loss function learns favorable embeddings compared to the standard InfoNCE whenever at least noisy ranking information can be obtained or when the definition of positives and negatives is blurry. We demonstrate this for a supervised classification task with additional superclass labels and noisy similarity scores. Furthermore, we show that RINCE can also be applied to unsupervised training with experiments on unsupervised representation learning from videos. In particular, the embedding yields higher classification accuracy, retrieval rates and performs better in out-of-distribution detection than the standard InfoNCE loss.