CASteer: Cross-Attention Steering for Controllable Concept Erasure
This work provides a practical, generalizable solution for controllable image generation, addressing the need for precise concept erasure without retraining.
CASteer introduces a training-free framework for concept erasure in diffusion models that uses steering vectors to dynamically suppress undesired concepts during inference, outperforming state-of-the-art methods while preserving unrelated content and image quality.
Diffusion models have transformed image generation, yet controlling their outputs to reliably erase undesired concepts remains challenging. Existing approaches usually require task-specific training and struggle to generalize across both concrete (e.g., objects) and abstract (e.g., styles) concepts. We propose CASteer (Cross-Attention Steering), a training-free framework for concept erasure in diffusion models using steering vectors to influence hidden representations dynamically. CASteer precomputes concept-specific steering vectors by averaging neural activations from images generated for each target concept. During inference, it dynamically applies these vectors to suppress undesired concepts only when they appear, ensuring that unrelated regions remain unaffected. This selective activation enables precise, context-aware erasure without degrading overall image quality. This approach achieves effective removal of harmful or unwanted content across a wide range of visual concepts, all without model retraining. CASteer outperforms state-of-the-art concept erasure techniques while preserving unrelated content and minimizing unintended effects.