Accelerating Conditional Prompt Learning via Masked Image Modeling for Vision-Language Models
This work addresses the challenge of efficient adaptation for vision-language models in real-world applications, offering a lightweight solution to mitigate overfitting, though it is incremental as it builds upon existing prompt learning methods like CoOp and CoCoOp.
The paper tackled the problem of overfitting in prompt learning for vision-language models, which limits generalization to unseen categories, and introduced ProMIM, a framework that integrates masked image modeling to enhance conditional prompt learning, resulting in improved generalization performance across zero-shot and few-shot classification tasks with negligible additional computational cost.
Vision-language models (VLMs) like CLIP excel in zero-shot learning but often require resource-intensive training to adapt to new tasks. Prompt learning techniques, such as CoOp and CoCoOp, offer efficient adaptation but tend to overfit to known classes, limiting generalization to unseen categories. We introduce ProMIM, a plug-and-play framework that enhances conditional prompt learning by integrating masked image modeling (MIM) into existing VLM pipelines. ProMIM leverages a simple yet effective masking strategy to generate robust, instance-conditioned prompts, seamlessly augmenting methods like CoOp and CoCoOp without altering their core architectures. By masking only visible image patches and using these representations to guide prompt generation, ProMIM improves feature robustness and mitigates overfitting, all while introducing negligible additional computational cost. Extensive experiments across zero-shot and few-shot classification tasks demonstrate that ProMIM consistently boosts generalization performance when plugged into existing approaches, providing a practical, lightweight solution for real-world vision-language applications.