Delta Sampling: Data-Free Knowledge Transfer Across Diffusion Models
This addresses a practical issue for users of open-source diffusion models by enabling plug-and-play knowledge transfer, though it is incremental as it builds on existing adaptation methods.
The paper tackles the problem of reusing adaptation components like LoRA and ControlNet across different versions of diffusion models (e.g., Stable Diffusion 1.x to 2.x) without access to original training data, and proposes Delta Sampling, which achieves consistent improvements in transferring visual styles, concepts, and structures.
Diffusion models like Stable Diffusion (SD) drive a vibrant open-source ecosystem including fully fine-tuned checkpoints and parameter-efficient adapters such as LoRA, LyCORIS, and ControlNet. However, these adaptation components are tightly coupled to a specific base model, making them difficult to reuse when the base model is upgraded (e.g., from SD 1.x to 2.x) due to substantial changes in model parameters and architecture. In this work, we propose Delta Sampling (DS), a novel method that enables knowledge transfer across base models with different architectures, without requiring access to the original training data. DS operates entirely at inference time by leveraging the delta: the difference in model predictions before and after the adaptation of a base model. This delta is then used to guide the denoising process of a new base model. We evaluate DS across various SD versions, demonstrating that DS achieves consistent improvements in creating desired effects (e.g., visual styles, semantic concepts, and structures) under different sampling strategies. These results highlight DS as an effective, plug-and-play mechanism for knowledge transfer in diffusion-based image synthesis. Code:~ https://github.com/Zhidong-Gao/DeltaSampling