Robust Contrastive Learning against Noisy Views
This addresses a critical issue for practitioners using contrastive learning in noisy real-world data scenarios, offering a simple drop-in replacement for existing losses.
The paper tackles the problem of contrastive learning producing suboptimal representations when positive pairs contain noisy views with no shared information, by proposing a robust contrastive loss function that provides consistent improvements over state-of-the-art methods on image, video, and graph benchmarks with real-world noise patterns.
Contrastive learning relies on an assumption that positive pairs contain related views, e.g., patches of an image or co-occurring multimodal signals of a video, that share certain underlying information about an instance. But what if this assumption is violated? The literature suggests that contrastive learning produces suboptimal representations in the presence of noisy views, e.g., false positive pairs with no apparent shared information. In this work, we propose a new contrastive loss function that is robust against noisy views. We provide rigorous theoretical justifications by showing connections to robust symmetric losses for noisy binary classification and by establishing a new contrastive bound for mutual information maximization based on the Wasserstein distance measure. The proposed loss is completely modality-agnostic and a simple drop-in replacement for the InfoNCE loss, which makes it easy to apply to existing contrastive frameworks. We show that our approach provides consistent improvements over the state-of-the-art on image, video, and graph contrastive learning benchmarks that exhibit a variety of real-world noise patterns.