ArCL: Enhancing Contrastive Learning with Augmentation-Robust Representations
This work addresses the problem of poor transferability in self-supervised learning for machine learning researchers, offering a novel theoretical framework and method, though it is incremental as it builds on existing contrastive learning algorithms.
The paper tackled the limited theoretical understanding of self-supervised contrastive learning's transferability by analyzing the impact of data augmentation, revealing that it fails to learn domain-invariant features. They proposed ArCL, a method that guarantees domain-invariant features and significantly improves transferability in experiments on several datasets.
Self-Supervised Learning (SSL) is a paradigm that leverages unlabeled data for model training. Empirical studies show that SSL can achieve promising performance in distribution shift scenarios, where the downstream and training distributions differ. However, the theoretical understanding of its transferability remains limited. In this paper, we develop a theoretical framework to analyze the transferability of self-supervised contrastive learning, by investigating the impact of data augmentation on it. Our results reveal that the downstream performance of contrastive learning depends largely on the choice of data augmentation. Moreover, we show that contrastive learning fails to learn domain-invariant features, which limits its transferability. Based on these theoretical insights, we propose a novel method called Augmentation-robust Contrastive Learning (ArCL), which guarantees to learn domain-invariant features and can be easily integrated with existing contrastive learning algorithms. We conduct experiments on several datasets and show that ArCL significantly improves the transferability of contrastive learning.