Do We Really Need a Large Number of Visual Prompts?
This work addresses computational overhead for resource-constrained edge devices by optimizing visual prompt tuning, though it is incremental as it builds on existing VPT methods.
The paper tackles the inefficiency of using many visual prompts in parameter-efficient transfer learning by showing that more prompts do not linearly improve performance, and proposes Prompt Condensation to reduce prompts by ~70% while maintaining accuracy on FGVC and VTAB-1k tasks.
Due to increasing interest in adapting models on resource-constrained edges, parameter-efficient transfer learning has been widely explored. Among various methods, Visual Prompt Tuning (VPT), prepending learnable prompts to input space, shows competitive fine-tuning performance compared to training of full network parameters. However, VPT increases the number of input tokens, resulting in additional computational overhead. In this paper, we analyze the impact of the number of prompts on fine-tuning performance and self-attention operation in a vision transformer architecture. Through theoretical and empirical analysis we show that adding more prompts does not lead to linear performance improvement. Further, we propose a Prompt Condensation (PC) technique that aims to prevent performance degradation from using a small number of prompts. We validate our methods on FGVC and VTAB-1k tasks and show that our approach reduces the number of prompts by ~70% while maintaining accuracy.