Efficient Fine-Tuning and Concept Suppression for Pruned Diffusion Models
This addresses safety and deployment issues for resource-constrained scenarios like mobile devices, offering an incremental improvement over existing pruning and distillation methods.
The paper tackles the problem of pruned diffusion models generating undesirable content like copyrighted material, even after fine-tuning, by proposing a bilevel optimization framework that unifies fine-tuning and unlearning to suppress such concepts while maintaining efficiency and style transfer capabilities.
Recent advances in diffusion generative models have yielded remarkable progress. While the quality of generated content continues to improve, these models have grown considerably in size and complexity. This increasing computational burden poses significant challenges, particularly in resource-constrained deployment scenarios such as mobile devices. The combination of model pruning and knowledge distillation has emerged as a promising solution to reduce computational demands while preserving generation quality. However, this technique inadvertently propagates undesirable behaviors, including the generation of copyrighted content and unsafe concepts, even when such instances are absent from the fine-tuning dataset. In this paper, we propose a novel bilevel optimization framework for pruned diffusion models that consolidates the fine-tuning and unlearning processes into a unified phase. Our approach maintains the principal advantages of distillation-namely, efficient convergence and style transfer capabilities-while selectively suppressing the generation of unwanted content. This plug-in framework is compatible with various pruning and concept unlearning methods, facilitating efficient, safe deployment of diffusion models in controlled environments.