Source-Prior-Driven Selective Adaptation for Efficient Diffusion Model Finetuning
For practitioners fine-tuning large diffusion models, this method offers an efficient way to adapt to new domains or styles while preserving the model's broad generative ability, addressing a key trade-off in fine-tuning.
This work proposes a source-prior-driven selective adaptation method for fine-tuning diffusion models that explicitly identifies and updates only parameters with minimal impact on general generative capability, achieving a better trade-off between target-specific generation and retention of pretrained knowledge than existing methods.
Fine-tuning large diffusion models for new domains or styles involves a trade-off: improving target-specific generation often degrades the pretrained model's broad generative capability. Existing full and parameter-efficient fine-tuning methods typically handle this trade-off only implicitly. In this work, we propose a novel source-prior-driven selective adaptation method to efficiently fine-tune diffusion models, achieving a favorable trade-off. Our method relies on two key observations: (1) the loss of general generative capability is highly inconsistent across pretrained parameters, and (2) parameters that have a relatively small impact on the model's general generative capability remain structurally inconsistent across layers and parameter types. Motivated by these observations, we first learn a static mask to explicitly identify parameters better suited for downstream adaptation, and then construct structured update strategies for the selected subset. Experiments show that our method achieves a better adaptation-retention trade-off than existing strong baselines.