Catastrophic Compositional Generation: Why Vanilla Diffusion Models Fail to Extrapolate
Identifies a fundamental limitation of diffusion models for compositional generation, relevant to researchers working on conditional generative models.
The paper argues that vanilla conditional diffusion models cannot efficiently perform compositional generation when the target distribution is out-of-distribution, showing that score estimation error catastrophically degrades performance even with inference-time corrections.
The task of compositional generation involves using a conditional generative model, trained only on a subset of the possible conditions, to produce samples from compositionally-defined target distributions such as a geometric combination of the source distributions. In this work, we argue that this task is often infeasible for vanilla conditional diffusion models: we conjecture that no inference-time technique can efficiently produce samples from the target distribution in certain well-motivated settings. This idea is supported by theory-guided generalization arguments and carefully-designed experiments on both synthetic and realistic data. In particular, while recent methods such as Feynman-Kac correction reduce inference-time approximation error, our results show that score estimation error has a more catastrophic effect on performance when the target distribution is out-of-distribution with respect to the sources, highlighting the need for a different approach to this task.