Concept Removal Guidance: Evidence-Calibrated Negative Guidance for Safe Diffusion Sampling
For practitioners deploying text-to-image diffusion models, CRG provides a more reliable inference-time safety mechanism that avoids the safety-fidelity trade-off of fixed negative guidance and is robust to adversarial prompts.
Concept Removal Guidance (CRG) is a training-free method that adaptively calibrates negative guidance during diffusion sampling to suppress unwanted concepts while preserving image fidelity. It reduces attack success rates on red-teaming benchmarks and extends to suppressing artist style and violence without fine-tuning.
Text-to-image diffusion models remain vulnerable to adversarial prompts that elicit disallowed content, motivating reliable inference-time controls. A popular approach is negative guidance, which subtracts a negative prompt direction with a fixed weight. However, it often forces a safety-fidelity trade-off, causing artifacts or prompt drift when over-applied and failing under attacks when under-applied. Dynamic variants reweight guidance using posterior-odds signals, which can be brittle for open-vocabulary compositional prompts, while lightweight similarity-based methods ignore the evolving image evidence along the denoising trajectory. We introduce Concept Removal Guidance (CRG), a training-free method that estimates unwanted-concept presence at each diffusion step from the model's noise predictions, and adaptively calibrates negative guidance via a closed-form constrained update enforcing a target presence threshold while minimally perturbing the conditional trajectory. Across red-teaming benchmarks, CRG reduces attack success rates while preserving benign fidelity, and extends to additional suppression targets such as artist style and violence without fine-tuning or external classifiers.