VPA: Fully Test-Time Visual Prompt Adaptation
This work addresses the need for efficient and robust adaptation of vision models in real-world scenarios like OOD generalization and domain adaptation, representing a novel extension of prompt tuning to visual domains with incremental advancements.
The paper tackles the problem of adapting visual models to downstream tasks without source-domain information by introducing Visual Prompt Adaptation (VPA), a framework that uses learnable tokens for test-time adaptation, resulting in improvements such as 3.3% in out-of-distribution generalization and 6.5% in corruption robustness.
Textual prompt tuning has demonstrated significant performance improvements in adapting natural language processing models to a variety of downstream tasks by treating hand-engineered prompts as trainable parameters. Inspired by the success of textual prompting, several studies have investigated the efficacy of visual prompt tuning. In this work, we present Visual Prompt Adaptation (VPA), the first framework that generalizes visual prompting with test-time adaptation. VPA introduces a small number of learnable tokens, enabling fully test-time and storage-efficient adaptation without necessitating source-domain information. We examine our VPA design under diverse adaptation settings, encompassing single-image, batched-image, and pseudo-label adaptation. We evaluate VPA on multiple tasks, including out-of-distribution (OOD) generalization, corruption robustness, and domain adaptation. Experimental results reveal that VPA effectively enhances OOD generalization by 3.3% across various models, surpassing previous test-time approaches. Furthermore, we show that VPA improves corruption robustness by 6.5% compared to strong baselines. Finally, we demonstrate that VPA also boosts domain adaptation performance by relatively 5.2%. Our VPA also exhibits marked effectiveness in improving the robustness of zero-shot recognition for vision-language models.