CPL: Counterfactual Prompt Learning for Vision and Language Models
This work addresses the challenge of improving generalization in few-shot learning for vision and language models, which is important for applications requiring efficient adaptation to new concepts with limited data, though it is incremental as it builds on existing prompt tuning methods.
The paper tackles the problem of spurious or entangled representations in prompt tuning for vision and language models like CLIP, which leads to poor generalization to unseen concepts, and presents CPL, a method that uses counterfactual generation and contrastive learning to achieve superior few-shot performance, with improvements such as 3.55% average relative improvement on unseen classes in image classification.
Prompt tuning is a new few-shot transfer learning technique that only tunes the learnable prompt for pre-trained vision and language models such as CLIP. However, existing prompt tuning methods tend to learn spurious or entangled representations, which leads to poor generalization to unseen concepts. Towards non-spurious and efficient prompt learning from limited examples, this paper presents a novel \underline{\textbf{C}}ounterfactual \underline{\textbf{P}}rompt \underline{\textbf{L}}earning (CPL) method for vision and language models, which simultaneously employs counterfactual generation and contrastive learning in a joint optimization framework. Particularly, CPL constructs counterfactual by identifying minimal non-spurious feature change between semantically-similar positive and negative samples that causes concept change, and learns more generalizable prompt representation from both factual and counterfactual examples via contrastive learning. Extensive experiments demonstrate that CPL can obtain superior few-shot performance on different vision and language tasks than previous prompt tuning methods on CLIP. On image classification, we achieve 3.55\% average relative improvement on unseen classes across seven datasets; on image-text retrieval and visual question answering, we gain up to 4.09\% and 25.08\% relative improvements across three few-shot scenarios on unseen test sets respectively.