Information-Theoretic Proofs for Diffusion Sampling
This work addresses the theoretical underpinnings of diffusion sampling for researchers in machine learning, offering a discrete-time analysis that is more accessible and precise than prior continuous-time approaches.
The paper tackles the problem of analyzing diffusion-based sampling methods for generative modeling by providing non-asymptotic convergence guarantees under broad assumptions, showing that with small step sizes and good conditional mean estimators, the sampling distribution is provably close to the target distribution.
This paper provides an elementary, self-contained analysis of diffusion-based sampling methods for generative modeling. In contrast to existing approaches that rely on continuous-time processes and then discretize, our treatment works directly with discrete-time stochastic processes and yields precise non-asymptotic convergence guarantees under broad assumptions. The key insight is to couple the sampling process of interest with an idealized comparison process that has an explicit Gaussian-convolution structure. We then leverage simple identities from information theory, including the I-MMSE relationship, to bound the discrepancy (in terms of the Kullback-Leibler divergence) between these two discrete-time processes. In particular, we show that, if the diffusion step sizes are chosen sufficiently small and one can approximate certain conditional mean estimators well, then the sampling distribution is provably close to the target distribution. Our results also provide a transparent view on how to accelerate convergence by using additional randomness in each step to match higher-order moments in the comparison process.