Hierarchical Refinement of Universal Multimodal Attacks on Vision-Language Models
This work addresses efficiency issues in adversarial attacks for vision-language models, but it is incremental as it builds on existing universal attack methods.
The paper tackles the computational overhead of sample-specific adversarial attacks on vision-language models by proposing a hierarchical refinement attack framework, achieving superior transferability across tasks, models, and datasets.
Existing adversarial attacks for VLP models are mostly sample-specific, resulting in substantial computational overhead when scaled to large datasets or new scenarios. To overcome this limitation, we propose Hierarchical Refinement Attack (HRA), a multimodal universal attack framework for VLP models. For the image modality, we refine the optimization path by leveraging a temporal hierarchy of historical and estimated future gradients to avoid local minima and stabilize universal perturbation learning. For the text modality, it hierarchically models textual importance by considering both intra- and inter-sentence contributions to identify globally influential words, which are then used as universal text perturbations. Extensive experiments across various downstream tasks, VLP models, and datasets, demonstrate the superior transferability of the proposed universal multimodal attacks.