A Statistical Theory of Contrastive Learning via Approximate Sufficient Statistics
This provides a theoretical foundation for contrastive learning, which is incremental as it extends prior work on CLIP to SimCLR and other methods.
The authors tackled the problem of understanding contrastive learning theoretically by developing a framework based on approximate sufficient statistics, showing that minimizing contrastive losses yields encoders that are approximately sufficient and can be adapted to downstream tasks with performance depending on sufficiency and augmentation error.
Contrastive learning -- a modern approach to extract useful representations from unlabeled data by training models to distinguish similar samples from dissimilar ones -- has driven significant progress in foundation models. In this work, we develop a new theoretical framework for analyzing data augmentation-based contrastive learning, with a focus on SimCLR as a representative example. Our approach is based on the concept of \emph{approximate sufficient statistics}, which we extend beyond its original definition in \cite{oko2025statistical} for contrastive language-image pretraining (CLIP) using KL-divergence. We generalize it to equivalent forms and general f-divergences, and show that minimizing SimCLR and other contrastive losses yields encoders that are approximately sufficient. Furthermore, we demonstrate that these near-sufficient encoders can be effectively adapted to downstream regression and classification tasks, with performance depending on their sufficiency and the error induced by data augmentation in contrastive learning. Concrete examples in linear regression and topic classification are provided to illustrate the broad applicability of our results.