Self-supervised Pre-training with Hard Examples Improves Visual Representations
This work provides an incremental improvement for researchers and practitioners working on self-supervised learning and visual representation, offering a method to enhance pre-trained model generalization.
This paper proposes a new data augmentation method, Hard Examples (HEXA), for self-supervised pre-training (SSP) that generates training examples with harder-to-predict pseudo-labels. When applied to MoCo-v2 and DeepCluster-v2, HEXA variants achieve new state-of-the-art results on various benchmarks, particularly when limited task supervision is available for fine-tuning.
Self-supervised pre-training (SSP) employs random image transformations to generate training data for visual representation learning. In this paper, we first present a modeling framework that unifies existing SSP methods as learning to predict pseudo-labels. Then, we propose new data augmentation methods of generating training examples whose pseudo-labels are harder to predict than those generated via random image transformations. Specifically, we use adversarial training and CutMix to create hard examples (HEXA) to be used as augmented views for MoCo-v2 and DeepCluster-v2, leading to two variants HEXA_{MoCo} and HEXA_{DCluster}, respectively. In our experiments, we pre-train models on ImageNet and evaluate them on multiple public benchmarks. Our evaluation shows that the two new algorithm variants outperform their original counterparts, and achieve new state-of-the-art on a wide range of tasks where limited task supervision is available for fine-tuning. These results verify that hard examples are instrumental in improving the generalization of the pre-trained models.