GIFT: Gradient-aware Immunization of diffusion models against malicious Fine-Tuning with safe concepts retention
This addresses the vulnerability of generative models to adversarial attacks, offering a solution for creating inherently safer AI systems, though it is incremental as it builds on existing immunization and fine-tuning defenses.
The paper tackles the problem of defending diffusion models against malicious fine-tuning attacks while retaining safe content generation, achieving robust resistance that significantly impairs re-learning of harmful concepts while maintaining safe generative quality.
We present GIFT: a {G}radient-aware {I}mmunization technique to defend diffusion models against malicious {F}ine-{T}uning while preserving their ability to generate safe content. Existing safety mechanisms like safety checkers are easily bypassed, and concept erasure methods fail under adversarial fine-tuning. GIFT addresses this by framing immunization as a bi-level optimization problem: the upper-level objective degrades the model's ability to represent harmful concepts using representation noising and maximization, while the lower-level objective preserves performance on safe data. GIFT achieves robust resistance to malicious fine-tuning while maintaining safe generative quality. Experimental results show that our method significantly impairs the model's ability to re-learn harmful concepts while maintaining performance on safe content, offering a promising direction for creating inherently safer generative models resistant to adversarial fine-tuning attacks.