Evolving Contextual Safety in Multi-Modal Large Language Models via Inference-Time Self-Reflective Memory
This work addresses safety risks in multi-modal AI systems, which is crucial for real-world deployment, though it appears incremental as it builds on existing safety research with a novel method.
The paper tackles the problem of contextual safety vulnerabilities in multi-modal large language models by introducing MM-SafetyBench++, a benchmark for evaluating contextual safety, and EchoSafe, a training-free framework that uses self-reflective memory to improve safety during inference. The result is that EchoSafe achieves superior performance on various multi-modal safety benchmarks, establishing a strong baseline for advancing contextual safety.
Multi-modal Large Language Models (MLLMs) have achieved remarkable performance across a wide range of visual reasoning tasks, yet their vulnerability to safety risks remains a pressing concern. While prior research primarily focuses on jailbreak defenses that detect and refuse explicitly unsafe inputs, such approaches often overlook contextual safety, which requires models to distinguish subtle contextual differences between scenarios that may appear similar but diverge significantly in safety intent. In this work, we present MM-SafetyBench++, a carefully curated benchmark designed for contextual safety evaluation. Specifically, for each unsafe image-text pair, we construct a corresponding safe counterpart through minimal modifications that flip the user intent while preserving the underlying contextual meaning, enabling controlled evaluation of whether models can adapt their safety behaviors based on contextual understanding. Further, we introduce EchoSafe, a training-free framework that maintains a self-reflective memory bank to accumulate and retrieve safety insights from prior interactions. By integrating relevant past experiences into current prompts, EchoSafe enables context-aware reasoning and continual evolution of safety behavior during inference. Extensive experiments on various multi-modal safety benchmarks demonstrate that EchoSafe consistently achieves superior performance, establishing a strong baseline for advancing contextual safety in MLLMs. All benchmark data and code are available at https://echosafe-mllm.github.io.