Towards Adversarial Attack on Vision-Language Pre-training Models
This addresses the safety and reliability of VLP models for real-world deployment, but it is incremental as it builds on existing adversarial attack research in a new multimodal context.
The paper tackled the adversarial robustness of vision-language pre-training (VLP) models by analyzing attack settings and proposing Co-Attack, a novel multimodal method that collectively attacks image and text modalities, achieving improved attack performances on various downstream tasks and models.
While vision-language pre-training model (VLP) has shown revolutionary improvements on various vision-language (V+L) tasks, the studies regarding its adversarial robustness remain largely unexplored. This paper studied the adversarial attack on popular VLP models and V+L tasks. First, we analyzed the performance of adversarial attacks under different settings. By examining the influence of different perturbed objects and attack targets, we concluded some key observations as guidance on both designing strong multimodal adversarial attack and constructing robust VLP models. Second, we proposed a novel multimodal attack method on the VLP models called Collaborative Multimodal Adversarial Attack (Co-Attack), which collectively carries out the attacks on the image modality and the text modality. Experimental results demonstrated that the proposed method achieves improved attack performances on different V+L downstream tasks and VLP models. The analysis observations and novel attack method hopefully provide new understanding into the adversarial robustness of VLP models, so as to contribute their safe and reliable deployment in more real-world scenarios. Code is available at https://github.com/adversarial-for-goodness/Co-Attack.