CLIP-CID: Efficient CLIP Distillation via Cluster-Instance Discrimination
This addresses the computational inefficiency of CLIP for researchers and practitioners by enabling more efficient model deployment, though it is incremental as it builds on existing distillation methods.
The paper tackles the problem of efficiently distilling knowledge from large vision-language models like CLIP to smaller models, proposing CLIP-CID which filters 43.7% of data to reduce bias and uses cluster-instance discrimination, achieving state-of-the-art performance on downstream tasks.
Contrastive Language-Image Pre-training (CLIP) has achieved excellent performance over a wide range of tasks. However, the effectiveness of CLIP heavily relies on a substantial corpus of pre-training data, resulting in notable consumption of computational resources. Although knowledge distillation has been widely applied in single modality models, how to efficiently expand knowledge distillation to vision-language foundation models with extensive data remains relatively unexplored. In this paper, we introduce CLIP-CID, a novel distillation mechanism that effectively transfers knowledge from a large vision-language foundation model to a smaller model. We initially propose a simple but efficient image semantic balance method to reduce transfer learning bias and improve distillation efficiency. This method filters out 43.7% of image-text pairs from the LAION400M while maintaining superior performance. After that, we leverage cluster-instance discrimination to facilitate knowledge transfer from the teacher model to the student model, thereby empowering the student model to acquire a holistic semantic comprehension of the pre-training data. Experimental results demonstrate that CLIP-CID achieves state-of-the-art performance on various downstream tasks including linear probe and zero-shot classification.