DA-VPT: Semantic-Guided Visual Prompt Tuning for Vision Transformers
This addresses parameter-efficient fine-tuning for vision transformers, which is important for adapting large models with limited resources, though it appears incremental as it builds on existing prompt tuning methods.
The paper tackles the problem of improving visual prompt tuning for vision transformers by investigating how prompt distributions affect fine-tuning performance, proposing DA-VPT to guide prompts using semantic information from class data. The result shows improved performance on recognition and segmentation benchmarks, enabling more effective and efficient fine-tuning.
Visual Prompt Tuning (VPT) has become a promising solution for Parameter-Efficient Fine-Tuning (PEFT) approach for Vision Transformer (ViT) models by partially fine-tuning learnable tokens while keeping most model parameters frozen. Recent research has explored modifying the connection structures of the prompts. However, the fundamental correlation and distribution between the prompts and image tokens remain unexplored. In this paper, we leverage metric learning techniques to investigate how the distribution of prompts affects fine-tuning performance. Specifically, we propose a novel framework, Distribution Aware Visual Prompt Tuning (DA-VPT), to guide the distributions of the prompts by learning the distance metric from their class-related semantic data. Our method demonstrates that the prompts can serve as an effective bridge to share semantic information between image patches and the class token. We extensively evaluated our approach on popular benchmarks in both recognition and segmentation tasks. The results demonstrate that our approach enables more effective and efficient fine-tuning of ViT models by leveraging semantic information to guide the learning of the prompts, leading to improved performance on various downstream vision tasks.