Diffusion Tuning: Transferring Diffusion Models via Chain of Forgetting
This work addresses the need for efficient adaptation of diffusion models in generative modeling, offering a method that improves performance and speed for downstream applications, though it appears incremental as it builds on existing fine-tuning techniques.
The paper tackles the problem of computationally expensive training for diffusion models by proposing Diff-Tuning, a transfer approach that leverages a chain of forgetting trend to adapt pre-trained models for downstream tasks, achieving a 26% improvement over standard fine-tuning and a 24% faster convergence speed for ControlNet.
Diffusion models have significantly advanced the field of generative modeling. However, training a diffusion model is computationally expensive, creating a pressing need to adapt off-the-shelf diffusion models for downstream generation tasks. Current fine-tuning methods focus on parameter-efficient transfer learning but overlook the fundamental transfer characteristics of diffusion models. In this paper, we investigate the transferability of diffusion models and observe a monotonous chain of forgetting trend of transferability along the reverse process. Based on this observation and novel theoretical insights, we present Diff-Tuning, a frustratingly simple transfer approach that leverages the chain of forgetting tendency. Diff-Tuning encourages the fine-tuned model to retain the pre-trained knowledge at the end of the denoising chain close to the generated data while discarding the other noise side. We conduct comprehensive experiments to evaluate Diff-Tuning, including the transfer of pre-trained Diffusion Transformer models to eight downstream generations and the adaptation of Stable Diffusion to five control conditions with ControlNet. Diff-Tuning achieves a 26% improvement over standard fine-tuning and enhances the convergence speed of ControlNet by 24%. Notably, parameter-efficient transfer learning techniques for diffusion models can also benefit from Diff-Tuning.