VLM-KD: Knowledge Distillation from VLM for Long-Tail Visual Recognition
This addresses the problem of imbalanced data in visual recognition for researchers and practitioners, though it is incremental as it builds on existing knowledge distillation methods.
The paper tackles long-tail visual recognition by distilling knowledge from an off-the-shelf vision-language model into a vision encoder, achieving state-of-the-art performance on benchmark datasets.
For visual recognition, knowledge distillation typically involves transferring knowledge from a large, well-trained teacher model to a smaller student model. In this paper, we introduce an effective method to distill knowledge from an off-the-shelf vision-language model (VLM), demonstrating that it provides novel supervision in addition to those from a conventional vision-only teacher model. Our key technical contribution is the development of a framework that generates novel text supervision and distills free-form text into a vision encoder. We showcase the effectiveness of our approach, termed VLM-KD, across various benchmark datasets, showing that it surpasses several state-of-the-art long-tail visual classifiers. To our knowledge, this work is the first to utilize knowledge distillation with text supervision generated by an off-the-shelf VLM and apply it to vanilla randomly initialized vision encoders.