Supervision Accelerates Pre-training in Contrastive Semi-Supervised Learning of Visual Representations
This work addresses the computational cost of pre-training in computer vision, offering a more efficient approach for researchers and practitioners, though it is incremental as it builds on existing contrastive learning methods.
The paper tackles the problem of inefficient contrastive learning for visual representations by introducing a semi-supervised loss, SuNCEt, which uses a small amount of supervised data during pre-training. The result is that SuNCEt matches the accuracy of previous methods on ImageNet while using less than half the pre-training time and compute.
We investigate a strategy for improving the efficiency of contrastive learning of visual representations by leveraging a small amount of supervised information during pre-training. We propose a semi-supervised loss, SuNCEt, based on noise-contrastive estimation and neighbourhood component analysis, that aims to distinguish examples of different classes in addition to the self-supervised instance-wise pretext tasks. On ImageNet, we find that SuNCEt can be used to match the semi-supervised learning accuracy of previous contrastive approaches while using less than half the amount of pre-training and compute. Our main insight is that leveraging even a small amount of labeled data during pre-training, and not only during fine-tuning, provides an important signal that can significantly accelerate contrastive learning of visual representations. Our code is available online at github.com/facebookresearch/suncet.