Contrasting Contrastive Self-Supervised Representation Learning Pipelines
This work provides a systematic benchmark for researchers to evaluate self-supervised learning, but it is incremental as it focuses on analysis rather than introducing new methods.
The paper analyzes contrastive self-supervised learning methods by examining over 700 experiments across 30 encoders, 4 datasets, and 20 downstream tasks to understand how training methods and datasets affect performance.
In the past few years, we have witnessed remarkable breakthroughs in self-supervised representation learning. Despite the success and adoption of representations learned through this paradigm, much is yet to be understood about how different training methods and datasets influence performance on downstream tasks. In this paper, we analyze contrastive approaches as one of the most successful and popular variants of self-supervised representation learning. We perform this analysis from the perspective of the training algorithms, pre-training datasets and end tasks. We examine over 700 training experiments including 30 encoders, 4 pre-training datasets and 20 diverse downstream tasks. Our experiments address various questions regarding the performance of self-supervised models compared to their supervised counterparts, current benchmarks used for evaluation, and the effect of the pre-training data on end task performance. Our Visual Representation Benchmark (ViRB) is available at: https://github.com/allenai/virb.