E^2VPT: An Effective and Efficient Approach for Visual Prompt Tuning
This addresses the challenge of adapting large transformer-based models for new tasks with reduced computational cost, though it is incremental as it builds on existing prompt tuning methods.
The paper tackled the problem of parameter-intensive fine-tuning for large-scale pretrained vision models by proposing E^2VPT, an approach that introduces learnable prompts and pruning to improve efficiency and effectiveness, achieving state-of-the-art results with low parameter usage (e.g., 0.32% on VTAB-1k).
As the size of transformer-based models continues to grow, fine-tuning these large-scale pretrained vision models for new tasks has become increasingly parameter-intensive. Parameter-efficient learning has been developed to reduce the number of tunable parameters during fine-tuning. Although these methods show promising results, there is still a significant performance gap compared to full fine-tuning. To address this challenge, we propose an Effective and Efficient Visual Prompt Tuning (E^2VPT) approach for large-scale transformer-based model adaptation. Specifically, we introduce a set of learnable key-value prompts and visual prompts into self-attention and input layers, respectively, to improve the effectiveness of model fine-tuning. Moreover, we design a prompt pruning procedure to systematically prune low importance prompts while preserving model performance, which largely enhances the model's efficiency. Empirical results demonstrate that our approach outperforms several state-of-the-art baselines on two benchmarks, with considerably low parameter usage (e.g., 0.32% of model parameters on VTAB-1k). Our code is available at https://github.com/ChengHan111/E2VPT.