Learning Visual Prompts for Guiding the Attention of Vision Transformers
This work addresses the need for automated and adaptable visual prompting in computer vision, offering a more flexible alternative to manual methods, though it is incremental as it builds on existing visual prompting concepts.
The paper tackles the problem of manually designing visual prompts to guide vision transformers' attention by proposing a method to learn these prompts automatically, achieving effective redirection of attention across various pre-trained encoders without annotations or fine-tuning.
Visual prompting infuses visual information into the input image to adapt models toward specific predictions and tasks. Recently, manually crafted markers such as red circles are shown to guide the model to attend to a target region on the image. However, these markers only work on models trained with data containing those markers. Moreover, finding these prompts requires guesswork or prior knowledge of the domain on which the model is trained. This work circumvents manual design constraints by proposing to learn the visual prompts for guiding the attention of vision transformers. The learned visual prompt, added to any input image would redirect the attention of the pre-trained vision transformer to its spatial location on the image. Specifically, the prompt is learned in a self-supervised manner without requiring annotations and without fine-tuning the vision transformer. Our experiments demonstrate the effectiveness of the proposed optimization-based visual prompting strategy across various pre-trained vision encoders.