Feedback-based Modal Mutual Search for Attacking Vision-Language Pre-training Models
This work addresses the security of cross-modal AI systems, which is crucial for real-world applications, by introducing a novel attack method that enhances transferability in black-box scenarios.
The paper tackles the vulnerability of vision-language pre-training models to adversarial attacks by proposing Feedback-based Modal Mutual Search (FMMS), which uses a modal mutual loss and target model feedback to generate adversarial examples, achieving significant improvements over state-of-the-art baselines on Flickr30K and MSCOCO datasets.
Although vision-language pre-training (VLP) models have achieved remarkable progress on cross-modal tasks, they remain vulnerable to adversarial attacks. Using data augmentation and cross-modal interactions to generate transferable adversarial examples on surrogate models, transfer-based black-box attacks have become the mainstream methods in attacking VLP models, as they are more practical in real-world scenarios. However, their transferability may be limited due to the differences on feature representation across different models. To this end, we propose a new attack paradigm called Feedback-based Modal Mutual Search (FMMS). FMMS introduces a novel modal mutual loss (MML), aiming to push away the matched image-text pairs while randomly drawing mismatched pairs closer in feature space, guiding the update directions of the adversarial examples. Additionally, FMMS leverages the target model feedback to iteratively refine adversarial examples, driving them into the adversarial region. To our knowledge, this is the first work to exploit target model feedback to explore multi-modality adversarial boundaries. Extensive empirical evaluations on Flickr30K and MSCOCO datasets for image-text matching tasks show that FMMS significantly outperforms the state-of-the-art baselines.