MAA: Meticulous Adversarial Attack against Vision-Language Pre-trained Models
This work addresses a key bottleneck in evaluating robustness for multi-modal AI systems, offering a more effective adversarial attack method for researchers and practitioners.
The paper tackles the limited transferability of adversarial attacks across vision-language pre-trained models by proposing MAA, which exploits model-independent vulnerabilities and achieves enhanced generalizability, as demonstrated through extensive experiments on diverse models and datasets.
Current adversarial attacks for evaluating the robustness of vision-language pre-trained (VLP) models in multi-modal tasks suffer from limited transferability, where attacks crafted for a specific model often struggle to generalize effectively across different models, limiting their utility in assessing robustness more broadly. This is mainly attributed to the over-reliance on model-specific features and regions, particularly in the image modality. In this paper, we propose an elegant yet highly effective method termed Meticulous Adversarial Attack (MAA) to fully exploit model-independent characteristics and vulnerabilities of individual samples, achieving enhanced generalizability and reduced model dependence. MAA emphasizes fine-grained optimization of adversarial images by developing a novel resizing and sliding crop (RScrop) technique, incorporating a multi-granularity similarity disruption (MGSD) strategy. Extensive experiments across diverse VLP models, multiple benchmark datasets, and a variety of downstream tasks demonstrate that MAA significantly enhances the effectiveness and transferability of adversarial attacks. A large cohort of performance studies is conducted to generate insights into the effectiveness of various model configurations, guiding future advancements in this domain.