Unified Vision and Language Prompt Learning
This addresses the problem of adapting large vision-language models efficiently for researchers and practitioners, though it is incremental as it builds on existing prompt tuning methods.
The paper tackles the inconsistent performance of unimodal prompt tuning methods in vision-language models by proposing Unified Prompt Tuning (UPT), which jointly optimizes prompts across modalities, achieving a better trade-off on few-shot learning and domain generalization benchmarks across over 11 datasets.
Prompt tuning, a parameter- and data-efficient transfer learning paradigm that tunes only a small number of parameters in a model's input space, has become a trend in the vision community since the emergence of large vision-language models like CLIP. We present a systematic study on two representative prompt tuning methods, namely text prompt tuning and visual prompt tuning. A major finding is that none of the unimodal prompt tuning methods performs consistently well: text prompt tuning fails on data with high intra-class visual variances while visual prompt tuning cannot handle low inter-class variances. To combine the best from both worlds, we propose a simple approach called Unified Prompt Tuning (UPT), which essentially learns a tiny neural network to jointly optimize prompts across different modalities. Extensive experiments on over 11 vision datasets show that UPT achieves a better trade-off than the unimodal counterparts on few-shot learning benchmarks, as well as on domain generalization benchmarks. Code and models will be released to facilitate future research.