Set-level Guidance Attack: Boosting Adversarial Transferability of Vision-Language Pre-training Models
This work addresses a security vulnerability in multimodal AI systems by improving transfer attacks, which is incremental but important for robustness evaluation.
The paper tackles the problem of low adversarial transferability in vision-language pre-training models by proposing the Set-level Guidance Attack, which enhances cross-modal interactions and alignment-preserving augmentation, resulting in significant improvements in attack success rates, such as up to 30.21% on image-text retrieval tasks.
Vision-language pre-training (VLP) models have shown vulnerability to adversarial examples in multimodal tasks. Furthermore, malicious adversaries can be deliberately transferred to attack other black-box models. However, existing work has mainly focused on investigating white-box attacks. In this paper, we present the first study to investigate the adversarial transferability of recent VLP models. We observe that existing methods exhibit much lower transferability, compared to the strong attack performance in white-box settings. The transferability degradation is partly caused by the under-utilization of cross-modal interactions. Particularly, unlike unimodal learning, VLP models rely heavily on cross-modal interactions and the multimodal alignments are many-to-many, e.g., an image can be described in various natural languages. To this end, we propose a highly transferable Set-level Guidance Attack (SGA) that thoroughly leverages modality interactions and incorporates alignment-preserving augmentation with cross-modal guidance. Experimental results demonstrate that SGA could generate adversarial examples that can strongly transfer across different VLP models on multiple downstream vision-language tasks. On image-text retrieval, SGA significantly enhances the attack success rate for transfer attacks from ALBEF to TCL by a large margin (at least 9.78% and up to 30.21%), compared to the state-of-the-art.