SAVeS: Steering Safety Judgments in Vision-Language Models via Semantic Cues
This work addresses a critical vulnerability in multimodal safety systems for real-world and embodied AI applications, though it is incremental in analyzing existing mechanisms.
The study tackled the problem of understanding what drives safety judgments in vision-language models (VLMs) by investigating whether these judgments can be steered using semantic cues without altering scene content, and found that safety decisions are highly sensitive to such cues, indicating reliance on learned associations rather than grounded visual understanding.
Vision-language models (VLMs) are increasingly deployed in real-world and embodied settings where safety decisions depend on visual context. However, it remains unclear which visual evidence drives these judgments. We study whether multimodal safety behavior in VLMs can be steered by simple semantic cues. We introduce a semantic steering framework that applies controlled textual, visual, and cognitive interventions without changing the underlying scene content. To evaluate these effects, we propose SAVeS, a benchmark for situational safety under semantic cues, together with an evaluation protocol that separates behavioral refusal, grounded safety reasoning, and false refusals. Experiments across multiple VLMs and an additional state-of-the-art benchmark show that safety decisions are highly sensitive to semantic cues, indicating reliance on learned visual-linguistic associations rather than grounded visual understanding. We further demonstrate that automated steering pipelines can exploit these mechanisms, highlighting a potential vulnerability in multimodal safety systems.