ArGue: Attribute-Guided Prompt Tuning for Vision-Language Models
This addresses the problem of distribution shifts in vision-language models for researchers and practitioners, representing a novel method rather than an incremental improvement.
The paper tackled the limitation of soft prompt tuning in handling distribution shifts for vision-language models by introducing Attribute-Guided Prompt Tuning (ArGue), which significantly outperformed state-of-the-art methods on novel class prediction and out-of-distribution generalization tasks.
Although soft prompt tuning is effective in efficiently adapting Vision-Language (V&L) models for downstream tasks, it shows limitations in dealing with distribution shifts. We address this issue with Attribute-Guided Prompt Tuning (ArGue), making three key contributions. 1) In contrast to the conventional approach of directly appending soft prompts preceding class names, we align the model with primitive visual attributes generated by Large Language Models (LLMs). We posit that a model's ability to express high confidence in these attributes signifies its capacity to discern the correct class rationales. 2) We introduce attribute sampling to eliminate disadvantageous attributes, thus only semantically meaningful attributes are preserved. 3) We propose negative prompting, explicitly enumerating class-agnostic attributes to activate spurious correlations and encourage the model to generate highly orthogonal probability distributions in relation to these negative features. In experiments, our method significantly outperforms current state-of-the-art prompt tuning methods on both novel class prediction and out-of-distribution generalization tasks.