On the Collapse Errors Induced by the Deterministic Sampler for Diffusion Models
This addresses a fundamental limitation in diffusion models for researchers and practitioners, though it is incremental as it builds on existing techniques.
The paper identifies collapse errors in ODE-based diffusion models, where deterministic samplers cause overly concentrated data, and introduces a metric to quantify this effect, showing it occurs across various settings with empirical evidence.
Despite the widespread adoption of deterministic samplers in diffusion models (DMs), their potential limitations remain largely unexplored. In this paper, we identify collapse errors, a previously unrecognized phenomenon in ODE-based diffusion sampling, where the sampled data is overly concentrated in local data space. To quantify this effect, we introduce a novel metric and demonstrate that collapse errors occur across a variety of settings. When investigating its underlying causes, we observe a see-saw effect, where score learning in low noise regimes adversely impacts the one in high noise regimes. This misfitting in high noise regimes, coupled with the dynamics of deterministic samplers, ultimately causes collapse errors. Guided by these insights, we apply existing techniques from sampling, training, and architecture to empirically support our explanation of collapse errors. This work provides intensive empirical evidence of collapse errors in ODE-based diffusion sampling, emphasizing the need for further research into the interplay between score learning and deterministic sampling, an overlooked yet fundamental aspect of diffusion models.