When Think-with-Image Meets Safety: What Determines Multimodal Jailbreak Robustness?
For developers of large vision-language models, this work identifies a promising design pattern (explicit image-tool interaction) to improve safety against jailbreak attacks, while highlighting the need for pipeline-specific safety evaluation.
The paper investigates how different think-with-image reasoning paradigms affect multimodal jailbreak robustness, finding that explicit image-tool interaction reduces jailbreak success rates by around 30% relative on average across models, and proposes an image-tool safety vector framework to explain this effect.
Think-with-image reasoning is emerging as a new inference paradigm for large vision-language models, but its safety implications remain poorly understood. Existing systems already span multiple process designs, including direct response generation, text-only prior turn, visual-state manipulation, and explicit external image-tool invocation. In this paper, we ask which of these evaluated paradigms improves multimodal jailbreak robustness, and why. Across multiple vision-language models, explicit image-tool interaction yields the lowest attack success rates in our experiments, reducing jailbreak success by around 30% relative on average across the evaluated models. This finding is initially surprising: ASR remains low even when the returned image-tool output is manually overridden or itself unsafe-looking, but returns near direct-answering levels under text-only prior turn controls. These results indicate that the lower ASR is not explained by benign returned-image semantics or by the textual image-tool trace alone. To explain the pattern, we introduce an image-tool safety vector framework that models image-tool invocation as a residual shift in hidden representations toward a safety-relevant direction. Representation-level analyses and activation interventions support this account. Overall, our results suggest that explicit image-tool interaction is a promising design pattern for improving jailbreak robustness, while also motivating pipeline-specific safety evaluation.