Contrastive learning, multi-view redundancy, and linear models
This provides a theoretical foundation for contrastive learning, addressing a gap in understanding for researchers in self-supervised learning, though it is incremental as it builds on existing empirical work.
The paper tackles the theoretical understanding of contrastive learning in multi-view settings, showing that linear functions of learned representations are nearly optimal for downstream tasks when views provide redundant label information.
Self-supervised learning is an empirically successful approach to unsupervised learning based on creating artificial supervised learning problems. A popular self-supervised approach to representation learning is contrastive learning, which leverages naturally occurring pairs of similar and dissimilar data points, or multiple views of the same data. This work provides a theoretical analysis of contrastive learning in the multi-view setting, where two views of each datum are available. The main result is that linear functions of the learned representations are nearly optimal on downstream prediction tasks whenever the two views provide redundant information about the label.