AMIA: Automatic Masking and Joint Intention Analysis Makes LVLMs Robust Jailbreak Defenders
This addresses the safety and robustness of LVLMs against adversarial attacks, offering a practical defense without retraining, though it is incremental as it builds on existing defense mechanisms.
The paper tackled the problem of defending Large Vision-Language Models (LVLMs) against jailbreak attacks by introducing AMIA, a lightweight, inference-only defense that automatically masks image patches and conducts joint intention analysis, improving defense success rates from an average of 52.4% to 81.7% with only a 2% average accuracy drop.
We introduce AMIA, a lightweight, inference-only defense for Large Vision-Language Models (LVLMs) that (1) Automatically Masks a small set of text-irrelevant image patches to disrupt adversarial perturbations, and (2) conducts joint Intention Analysis to uncover and mitigate hidden harmful intents before response generation. Without any retraining, AMIA improves defense success rates across diverse LVLMs and jailbreak benchmarks from an average of 52.4% to 81.7%, preserves general utility with only a 2% average accuracy drop, and incurs only modest inference overhead. Ablation confirms both masking and intention analysis are essential for a robust safety-utility trade-off.