Diffusion models under low-noise regime
This work addresses gaps in understanding generative model reliability for practical applications with small perturbations, though it is incremental as it builds on existing regime theories.
The paper investigated diffusion models under low-noise conditions, revealing that models trained on disjoint data diverge near the data manifold despite converging in high-noise settings, with insights into how training set size and data geometry affect denoising trajectories.
Recent work on diffusion models proposed that they operate in two regimes: memorization, in which models reproduce their training data, and generalization, in which they generate novel samples. While this has been tested in high-noise settings, the behavior of diffusion models as effective denoisers when the corruption level is small remains unclear. To address this gap, we systematically investigated the behavior of diffusion models under low-noise diffusion dynamics, with implications for model robustness and interpretability. Using (i) CelebA subsets of varying sample sizes and (ii) analytic Gaussian mixture benchmarks, we reveal that models trained on disjoint data diverge near the data manifold even when their high-noise outputs converge. We quantify how training set size, data geometry, and model objective choice shape denoising trajectories and affect score accuracy, providing insights into how these models actually learn representations of data distributions. This work starts to address gaps in our understanding of generative model reliability in practical applications where small perturbations are common.