Explicit Uncertainty Modeling for Active CLIP Adaptation with Dual Prompt Tuning
This work addresses the problem of efficient adaptation of vision-language models for downstream tasks with limited labeled data, representing an incremental improvement in active learning techniques.
The paper tackles the challenge of adapting CLIP to image classification tasks with limited annotations by proposing an active learning method that uses dual-prompt tuning to explicitly model uncertainty, resulting in consistent outperformance over existing methods under the same annotation budget.
Pre-trained vision-language models such as CLIP exhibit strong transferability, yet adapting them to downstream image classification tasks under limited annotation budgets remains challenging. In active learning settings, the model must select the most informative samples for annotation from a large pool of unlabeled data. Existing approaches typically estimate uncertainty via entropy-based criteria or representation clustering, without explicitly modeling uncertainty from the model perspective. In this work, we propose a robust uncertainty modeling framework for active CLIP adaptation based on dual-prompt tuning. We introduce two learnable prompts in the textual branch of CLIP. The positive prompt enhances the discriminability of task-specific textual embeddings corresponding to light-weight tuned visual embeddings, improving classification reliability. Meanwhile, the negative prompt is trained in an reversed manner to explicitly model the probability that the predicted label is correct, providing a principled uncertainty signal for guiding active sample selection. Extensive experiments across different fine-tuning paradigms demonstrate that our method consistently outperforms existing active learning methods under the same annotation budget.