Is Your Conditional Diffusion Model Actually Denoising?
This work addresses a fundamental issue in conditional diffusion models, which are widely used in generative image and control tasks, by identifying and analyzing a systematic deviation that affects their reliability and performance.
The paper investigates the inductive biases of conditional diffusion models, revealing that their conditional generations consistently deviate from the idealized denoising process, causing disagreement among sampling algorithms, and introduces Schedule Deviation as a measure to quantify this deviation, showing it occurs regardless of model capacity or training data.
We study the inductive biases of diffusion models with a conditioning-variable, which have seen widespread application as both text-conditioned generative image models and observation-conditioned continuous control policies. We observe that when these models are queried conditionally, their generations consistently deviate from the idealized "denoising" process upon which diffusion models are formulated, inducing disagreement between popular sampling algorithms (e.g. DDPM, DDIM). We introduce Schedule Deviation, a rigorous measure which captures the rate of deviation from a standard denoising process, and provide a methodology to compute it. Crucially, we demonstrate that the deviation from an idealized denoising process occurs irrespective of the model capacity or amount of training data. We posit that this phenomenon occurs due to the difficulty of bridging distinct denoising flows across different parts of the conditioning space and show theoretically how such a phenomenon can arise through an inductive bias towards smoothness.