Challenging Vision-Language Models with Physically Deployable Multimodal Semantic Lighting Attacks
This work exposes a previously overlooked robustness gap in VLMs by demonstrating vulnerability to physically realizable semantic attacks, highlighting the need for physical-world security evaluation.
The authors propose Multimodal Semantic Lighting Attacks (MSLA), the first physically deployable adversarial attack against Vision-Language Models (VLMs), using controllable lighting to disrupt semantic alignment. In experiments, MSLA degrades zero-shot classification of CLIP variants and induces semantic hallucinations in LLaVA and BLIP across captioning and VQA tasks, demonstrating effectiveness and transferability in both digital and physical settings.
Vision-Language Models (VLMs) have shown remarkable performance, yet their security remains insufficiently understood. Existing adversarial studies focus almost exclusively on the digital setting, leaving physical-world threats largely unexplored. As VLMs are increasingly deployed in real environments, this gap becomes critical, since adversarial perturbations must be physically realizable. Despite this practical relevance, physical attacks against VLMs have not been systematically studied. Such attacks may induce recognition failures and further disrupt multimodal reasoning, leading to severe semantic misinterpretation in downstream tasks. Therefore, investigating physical attacks on VLMs is essential for assessing their real-world security risks. To address this gap, we propose Multimodal Semantic Lighting Attacks (MSLA), the first physically deployable adversarial attack framework against VLMs. MSLA uses controllable adversarial lighting to disrupt multimodal semantic understanding in real scenes, attacking semantic alignment rather than only task-specific outputs. Consequently, it degrades zero-shot classification performance of mainstream CLIP variants while inducing severe semantic hallucinations in advanced VLMs such as LLaVA and BLIP across image captioning and visual question answering (VQA). Extensive experiments in both digital and physical domains demonstrate that MSLA is effective, transferable, and practically realizable. Our findings provide the first evidence that VLMs are highly vulnerable to physically deployable semantic attacks, exposing a previously overlooked robustness gap and underscoring the urgent need for physical-world robustness evaluation of VLMs.