The Power of Contrast for Feature Learning: A Theoretical Analysis
This provides theoretical support for empirical findings in self-supervised learning, addressing a gap for researchers in machine learning, though it is incremental as it builds on existing contrastive learning frameworks.
The paper tackles the limited theoretical understanding of contrastive learning's superiority by proving that it outperforms autoencoders and GANs in feature recovery and in-domain tasks under linear settings, and shows that labeled data in supervised contrastive learning improves in-domain performance but harms transfer learning.
Contrastive learning has achieved state-of-the-art performance in various self-supervised learning tasks and even outperforms its supervised counterpart. Despite its empirical success, theoretical understanding of the superiority of contrastive learning is still limited. In this paper, under linear representation settings, (i) we provably show that contrastive learning outperforms the standard autoencoders and generative adversarial networks, two classical generative unsupervised learning methods, for both feature recovery and in-domain downstream tasks; (ii) we also illustrate the impact of labeled data in supervised contrastive learning. This provides theoretical support for recent findings that contrastive learning with labels improves the performance of learned representations in the in-domain downstream task, but it can harm the performance in transfer learning. We verify our theory with numerical experiments.