Establishing a stronger baseline for lightweight contrastive models
This work addresses the problem of inefficient training for lightweight contrastive models in computer vision, offering a practical solution for resource-constrained applications, though it is incremental as it builds on existing contrastive learning methods.
The paper tackled performance degradation in self-supervised contrastive learning for lightweight models like MobileNet and EfficientNet by proposing a stronger baseline without a pretrained teacher, improving linear evaluation accuracy from 36.3% to 62.3% for MobileNet-V3-Large and from 42.2% to 65.8% for EfficientNet-B0 on ImageNet.
Recent research has reported a performance degradation in self-supervised contrastive learning for specially designed efficient networks, such as MobileNet and EfficientNet. A common practice to address this problem is to introduce a pretrained contrastive teacher model and train the lightweight networks with distillation signals generated by the teacher. However, it is time and resource consuming to pretrain a teacher model when it is not available. In this work, we aim to establish a stronger baseline for lightweight contrastive models without using a pretrained teacher model. Specifically, we show that the optimal recipe for efficient models is different from that of larger models, and using the same training settings as ResNet50, as previous research does, is inappropriate. Additionally, we observe a common issu e in contrastive learning where either the positive or negative views can be noisy, and propose a smoothed version of InfoNCE loss to alleviate this problem. As a result, we successfully improve the linear evaluation results from 36.3\% to 62.3\% for MobileNet-V3-Large and from 42.2\% to 65.8\% for EfficientNet-B0 on ImageNet, closing the accuracy gap to ResNet50 with $5\times$ fewer parameters. We hope our research will facilitate the usage of lightweight contrastive models.