Why Instruction-Based Unlearning Fails in Diffusion Models?
This reveals a fundamental limitation of prompt-level instruction in diffusion models for researchers and practitioners working on model editing and safety.
The paper investigated whether instruction-based unlearning, effective for large language models, works for diffusion-based image generation models, and found through controlled experiments that diffusion models systematically fail to suppress targeted concepts when guided solely by natural-language unlearning instructions, with targeted concept representations persisting throughout generation.
Instruction-based unlearning has proven effective for modifying the behavior of large language models at inference time, but whether this paradigm extends to other generative models remains unclear. In this work, we investigate instruction-based unlearning in diffusion-based image generation models and show, through controlled experiments across multiple concepts and prompt variants, that diffusion models systematically fail to suppress targeted concepts when guided solely by natural-language unlearning instructions. By analyzing both the CLIP text encoder and cross-attention dynamics during the denoising process, we find that unlearning instructions do not induce sustained reductions in attention to the targeted concept tokens, causing the targeted concept representations to persist throughout generation. These results reveal a fundamental limitation of prompt-level instruction in diffusion models and suggest that effective unlearning requires interventions beyond inference-time language control.