Entropy-Based Dimension-Free Convergence and Loss-Adaptive Schedules for Diffusion Models
This work addresses convergence analysis and scheduling efficiency for diffusion models, offering a novel theoretical framework and practical improvement, though it is incremental relative to existing methods.
The paper tackles the problem of dimension-dependent convergence rates in diffusion models by developing an information-theoretic approach that achieves dimension-free convergence under mild assumptions, bounding KL divergence by O(H^2/K) where H is Shannon entropy and K is steps, and proposes a Loss-Adaptive Schedule (LAS) that improves sampling quality empirically.
Diffusion generative models synthesize samples by discretizing reverse-time dynamics driven by a learned score (or denoiser). Existing convergence analyses of diffusion models typically scale at least linearly with the ambient dimension, and sharper rates often depend on intrinsic-dimension assumptions or other geometric restrictions on the target distribution. We develop an alternative, information-theoretic approach to dimension-free convergence that avoids any geometric assumptions. Under mild assumptions on the target distribution, we bound KL divergence between the target and generated distributions by $O(H^2/K)$ (up to endpoint factors), where $H$ is the Shannon entropy and $K$ is the number of sampling steps. Moreover, using a reformulation of the KL divergence, we propose a Loss-Adaptive Schedule (LAS) for efficient discretization of reverse SDE which is lightweight and relies only on the training loss, requiring no post-training heavy computation. Empirically, LAS improves sampling quality over common heuristic schedules.