FALIP: Visual Prompt as Foveal Attention Boosts CLIP Zero-Shot Performance
This work addresses the issue of visual prompts degrading CLIP's zero-shot performance for tasks requiring precise image information, offering an incremental improvement over prior methods.
The paper tackles the problem of CLIP's zero-shot performance being hindered by manually designed visual prompts that alter image information, and proposes FALIP, a train-free method that inserts foveal attention masks into CLIP's attention module, boosting performance in tasks like referring expressions comprehension and image classification, with experimental results showing it outperforms existing methods on most metrics.
CLIP has achieved impressive zero-shot performance after pre-training on a large-scale dataset consisting of paired image-text data. Previous works have utilized CLIP by incorporating manually designed visual prompts like colored circles and blur masks into the images to guide the model's attention, showing enhanced zero-shot performance in downstream tasks. Although these methods have achieved promising results, they inevitably alter the original information of the images, which can lead to failure in specific tasks. We propose a train-free method Foveal-Attention CLIP (FALIP), which adjusts the CLIP's attention by inserting foveal attention masks into the multi-head self-attention module. We demonstrate FALIP effectively boosts CLIP zero-shot performance in tasks such as referring expressions comprehension, image classification, and 3D point cloud recognition. Experimental results further show that FALIP outperforms existing methods on most metrics and can augment current methods to enhance their performance.