In-Context Experience Replay Facilitates Safety Red-Teaming of Text-to-Image Diffusion Models
This work addresses the critical need for systematic safety evaluation tools in text-to-image models, which is important for developers and researchers to prevent harmful content generation.
The authors tackled the problem of evaluating safety mechanisms in text-to-image models by proposing ICER, a red-teaming framework that uses LLMs and bandit optimization to generate problematic prompts, which significantly outperformed existing methods in identifying vulnerabilities while maintaining high semantic similarity.
Text-to-image (T2I) models have shown remarkable progress, but their potential to generate harmful content remains a critical concern in the ML community. While various safety mechanisms have been developed, the field lacks systematic tools for evaluating their effectiveness against real-world misuse scenarios. In this work, we propose ICER, a novel red-teaming framework that leverages Large Language Models (LLMs) and a bandit optimization-based algorithm to generate interpretable and semantic meaningful problematic prompts by learning from past successful red-teaming attempts. Our ICER efficiently probes safety mechanisms across different T2I models without requiring internal access or additional training, making it broadly applicable to deployed systems. Through extensive experiments, we demonstrate that ICER significantly outperforms existing prompt attack methods in identifying model vulnerabilities while maintaining high semantic similarity with intended content. By uncovering that successful jailbreaking instances can systematically facilitate the discovery of new vulnerabilities, our work provides crucial insights for developing more robust safety mechanisms in T2I systems.