Concept Replacer: Replacing Sensitive Concepts in Diffusion Models via Precision Localization
This addresses the issue of unwanted content generation in diffusion models for users and developers, offering a targeted solution that is incremental over existing concept removal methods.
The paper tackles the problem of diffusion models generating unwanted content like explicit or violent images by proposing a method to precisely replace sensitive concepts without affecting non-target areas, achieving superior localization precision and coherent replacement compared to existing approaches.
As large-scale diffusion models continue to advance, they excel at producing high-quality images but often generate unwanted content, such as sexually explicit or violent content. Existing methods for concept removal generally guide the image generation process but can unintentionally modify unrelated regions, leading to inconsistencies with the original model. We propose a novel approach for targeted concept replacing in diffusion models, enabling specific concepts to be removed without affecting non-target areas. Our method introduces a dedicated concept localizer for precisely identifying the target concept during the denoising process, trained with few-shot learning to require minimal labeled data. Within the identified region, we introduce a training-free Dual Prompts Cross-Attention (DPCA) module to substitute the target concept, ensuring minimal disruption to surrounding content. We evaluate our method on concept localization precision and replacement efficiency. Experimental results demonstrate that our method achieves superior precision in localizing target concepts and performs coherent concept replacement with minimal impact on non-target areas, outperforming existing approaches.