Improving Adversarial Transferability in MLLMs via Dynamic Vision-Language Alignment Attack
This addresses a security vulnerability in MLLMs by improving adversarial transferability, which is incremental as it builds on existing attack methods but focuses on the vision-language modality alignment bottleneck.
The paper tackles the limited transferability of adversarial attacks across multimodal large language models (MLLMs) by introducing the Dynamic Vision-Language Alignment (DynVLA) Attack, which injects dynamic perturbations into the vision-language connector to enhance generalization, resulting in significant improvements in transferability across various models including BLIP2, InstructBLIP, MiniGPT4, LLaVA, and Gemini.
Multimodal Large Language Models (MLLMs), built upon LLMs, have recently gained attention for their capabilities in image recognition and understanding. However, while MLLMs are vulnerable to adversarial attacks, the transferability of these attacks across different models remains limited, especially under targeted attack setting. Existing methods primarily focus on vision-specific perturbations but struggle with the complex nature of vision-language modality alignment. In this work, we introduce the Dynamic Vision-Language Alignment (DynVLA) Attack, a novel approach that injects dynamic perturbations into the vision-language connector to enhance generalization across diverse vision-language alignment of different models. Our experimental results show that DynVLA significantly improves the transferability of adversarial examples across various MLLMs, including BLIP2, InstructBLIP, MiniGPT4, LLaVA, and closed-source models such as Gemini.