EMMA: Concept Erasure Benchmark with Comprehensive Semantic Metrics and Diverse Categories
This addresses the need for better evaluation of concept erasure methods to mitigate privacy, bias, and copyright issues in AI models, but it is incremental as it focuses on benchmarking rather than proposing new methods.
The paper tackles the problem of evaluating concept erasure techniques in text-to-image generation by introducing EMMA, a benchmark that tests five dimensions over 12 metrics, revealing that existing methods struggle with implicit prompts and visually similar non-target concepts, and some amplify bias.
The widespread adoption of text-to-image (T2I) generation has raised concerns about privacy, bias, and copyright violations. Concept erasure techniques offer a promising solution by selectively removing undesired concepts from pre-trained models without requiring full retraining. However, these methods are often evaluated on a limited set of concepts, relying on overly simplistic and direct prompts. To test the boundaries of concept erasure techniques, and assess whether they truly remove targeted concepts from model representations, we introduce EMMA, a benchmark that evaluates five key dimensions of concept erasure over 12 metrics. EMMA goes beyond standard metrics like image quality and time efficiency, testing robustness under challenging conditions, including indirect descriptions, visually similar non-target concepts, and potential gender and ethnicity bias, providing a socially aware analysis of method behavior. Using EMMA, we analyze five concept erasure methods across five domains (objects, celebrities, art styles, NSFW, and copyright). Our results show that existing methods struggle with implicit prompts (i.e., generating the erased concept when it is indirectly referenced) and visually similar non-target concepts (i.e., failing to generate non-targeted concepts resembling the erased one), while some amplify gender and ethnicity bias compared to the original model.