Domain Aligned CLIP for Few-shot Classification
This work addresses the need for efficient and robust adaptation of vision-language models like CLIP for downstream tasks, offering a sample-efficient method that avoids compute-intensive fine-tuning, though it is incremental as it builds on existing CLIP frameworks.
The paper tackled the problem of enhancing CLIP's few-shot classification performance without full fine-tuning by improving both intra-modal and inter-modal alignment, achieving a 2.3% improvement in 16-shot classification across 11 tasks and competitive results on out-of-distribution robustness benchmarks.
Large vision-language representation learning models like CLIP have demonstrated impressive performance for zero-shot transfer to downstream tasks while largely benefiting from inter-modal (image-text) alignment via contrastive objectives. This downstream performance can further be enhanced by full-scale fine-tuning which is often compute intensive, requires large labelled data, and can reduce out-of-distribution (OOD) robustness. Furthermore, sole reliance on inter-modal alignment might overlook the rich information embedded within each individual modality. In this work, we introduce a sample-efficient domain adaptation strategy for CLIP, termed Domain Aligned CLIP (DAC), which improves both intra-modal (image-image) and inter-modal alignment on target distributions without fine-tuning the main model. For intra-modal alignment, we introduce a lightweight adapter that is specifically trained with an intra-modal contrastive objective. To improve inter-modal alignment, we introduce a simple framework to modulate the precomputed class text embeddings. The proposed few-shot fine-tuning framework is computationally efficient, robust to distribution shifts, and does not alter CLIP's parameters. We study the effectiveness of DAC by benchmarking on 11 widely used image classification tasks with consistent improvements in 16-shot classification upon strong baselines by about 2.3% and demonstrate competitive performance on 4 OOD robustness benchmarks.