Revisiting Theory of Contrastive Learning for Domain Generalization
This work addresses domain generalization challenges in contrastive learning for real-world applications where downstream tasks may have distributional shifts or new labels, providing theoretical insights but is incremental as it extends existing theory.
The paper tackles the problem of contrastive learning's theoretical limitations in domain generalization by introducing novel generalization bounds that account for domain shift and new label spaces, revealing how performance depends on statistical discrepancies between pretraining and downstream distributions.
Contrastive learning is among the most popular and powerful approaches for self-supervised representation learning, where the goal is to map semantically similar samples close together while separating dissimilar ones in the latent space. Existing theoretical methods assume that downstream task classes are drawn from the same latent class distribution used during the pretraining phase. However, in real-world settings, downstream tasks may not only exhibit distributional shifts within the same label space but also introduce new or broader label spaces, leading to domain generalization challenges. In this work, we introduce novel generalization bounds that explicitly account for both types of mismatch: domain shift and domain generalization. Specifically, we analyze scenarios where downstream tasks either (i) draw classes from the same latent class space but with shifted distributions, or (ii) involve new label spaces beyond those seen during pretraining. Our analysis reveals how the performance of contrastively learned representations depends on the statistical discrepancy between pretraining and downstream distributions. This extended perspective allows us to derive provable guarantees on the performance of learned representations on average classification tasks involving class distributions outside the pretraining latent class set.