When Diffusion Model Can Ignore Dimension: An Entropy-Based Theory
Provides theoretical justification for why diffusion models remain efficient in high dimensions when data has compact latent structure, addressing a key gap between theory and practice.
The paper proves that for Gaussian mixture targets, diffusion model discretization error is controlled by the Shannon entropy of the latent mixture component rather than ambient dimension, with step complexity scaling linearly with latent entropy and logarithmically with data second moment. This extends to discrete targets where complexity depends on target entropy.
Diffusion models perform remarkably well on high-dimensional data such as images, often using only a modest number of reverse-time steps. Despite this practical success, existing convergence theory does not fully explain why such samplers remain efficient in high dimensions. Many prior KL guarantees bound the discretization error in terms of the ambient dimension, while other improved results replace this dependence using intrinsic-dimensional or geometric structure assumptions. In this work, we develop an alternative information-theoretic perspective on diffusion sampler convergence. We prove that, for Gaussian mixture targets, the discretization error is controlled by the Shannon entropy of the latent mixture component rather than by the ambient dimension. Consequently, the leading step complexity scales linearly with latent entropy and depends only logarithmically on the second moment of the data. Our analysis also extends to discrete target distributions, where the relevant complexity is the entropy of the target rather than the dimension of the embedding space. These results suggest that diffusion sampling can remain efficient in high-dimensional spaces when the data distribution admits a compact latent representation, as is widely believed to be the case for natural images.