A Sentence Speaks a Thousand Images: Domain Generalization through Distilling CLIP with Language Guidance
This addresses the problem of improving model robustness to unseen domains for machine learning practitioners, representing an incremental advance by applying knowledge distillation with vision-language models to domain generalization.
The paper tackles domain generalization by training a smaller model using a CLIP teacher model, with a new regularization method that aligns image representations with text embeddings, and shows that RISE outperforms state-of-the-art methods on benchmark datasets.
Domain generalization studies the problem of training a model with samples from several domains (or distributions) and then testing the model with samples from a new, unseen domain. In this paper, we propose a novel approach for domain generalization that leverages recent advances in large vision-language models, specifically a CLIP teacher model, to train a smaller model that generalizes to unseen domains. The key technical contribution is a new type of regularization that requires the student's learned image representations to be close to the teacher's learned text representations obtained from encoding the corresponding text descriptions of images. We introduce two designs of the loss function, absolute and relative distance, which provide specific guidance on how the training process of the student model should be regularized. We evaluate our proposed method, dubbed RISE (Regularized Invariance with Semantic Embeddings), on various benchmark datasets and show that it outperforms several state-of-the-art domain generalization methods. To our knowledge, our work is the first to leverage knowledge distillation using a large vision-language model for domain generalization. By incorporating text-based information, RISE improves the generalization capability of machine learning models.