On Success and Simplicity: A Second Look at Transferable Vision-Language Attack Pipeline
For researchers in adversarial attacks on VLPMs, this work simplifies the attack pipeline and achieves better performance, highlighting that domain knowledge is more important than complex operations.
The paper identifies three overlooked issues in transferable attacks on Vision-Language Pre-training Models (VLPMs) and proposes a simpler pipeline (SimVLA) that improves transferability and efficiency. On Flickr30k, SimVLA outperforms SOTA by 8.01%-14.71% in R@1 transferability while using only 35.73% of the time and 46.26% of the max VRAM.
Vision-Language Pre-training Models (VLPMs) are known to be vulnerable to adversarial attacks. Recent transferable attacks on VLPMs have followed a common pipeline with complicated loss functions or multi-stage text/image attacks. However, in this paper, we demonstrate that such a sophisticated attack pipeline can be simpler yet more successful. Specifically, we identify three previously overlooked issues caused by inappropriate cross-modal interactions and excessive operations. To address them, we propose the Simple Vision-Language Attack (SimVLA) pipeline, which observably improves transferability and efficiency. Experiments on four datasets and three downstream tasks validate the superiority of our pipeline. For instance, on Flickr30k text-image retrieval dataset, our SimVLA outperforms the SOTA baseline in R@1 transferability by 8.01\%-14.71\%, while consuming only about 35.73\% of the time and 46.26\% of the max VRAM. Overall, the superiority of our SimVLA highlights the importance of leveraging domain knowledge (e.g., our proposed cross-modal word identification), while blindly pursuing intricate operations (e.g, complex loss functions and redundant multi-stage designs) may even be harmful. We hope our SimVLA can serve as a simple yet effective backbone for future extensions. Code is available at https://github.com/RYC-98/SimVLA.