Enhancing Vision-Language Models Generalization via Diversity-Driven Novel Feature Synthesis
This addresses the challenge of maintaining generalization in vision-language models for downstream applications, though it is incremental as it builds on existing fine-tuning strategies.
The paper tackles the problem of vision-language models like CLIP overfitting and losing generalization when fine-tuned on downstream datasets, by proposing LDFS, a plug-and-play feature synthesis method that synthesizes new domain features without collecting additional data, resulting in improved generalization on unseen domains as shown in experiments.
Vision-language foundation models like CLIP have shown impressive zero-shot generalization, but finetuning on downstream datasets can cause overfitting and loss of its generalization ability on unseen domains. Although collecting additional data from new domains of interest is possible, this method is often impractical due to the challenges in obtaining annotated data. To address this, we propose a plug-and-play feature synthesis method called LDFS (Language-Guided Diverse Feature Synthesis) to synthesize new domain features and improve existing CLIP fine-tuning strategies. LDFS has three main contributions: 1) To synthesize novel domain features and promote diversity, we propose an instance-conditional feature augmentation strategy based on a text-guided feature augmentation loss. 2) To maintain feature quality after augmenting, we introduce a pairwise regularizer to preserve augmented feature coherence within the CLIP feature space. 3) We propose to use stochastic text feature augmentation to reduce the modality gap and further facilitate the process of text-guided feature synthesis. Extensive experiments show LDFS superiority in improving CLIP generalization ability on unseen domains without collecting data from those domains. The code will be made publicly available.