Stochstic Sampling for Generative Diffusion Models: From Euler-Maruyama to Higher-Order Schemes
This work provides a more rigorous and general theoretical framework for analyzing the convergence of stochastic samplers in generative diffusion models, which is important for researchers and practitioners developing and applying these models. It also introduces a sampler with provably higher convergence order, addressing a key limitation of existing methods.
This paper analyzes the convergence of generative diffusion models by considering initialization, score-matching, and discretization errors. It introduces a general strong scheme framework and proves a convergence theorem in the 2-Wasserstein distance, which is then applied to Euler-Maruyama, exponential integrators, and a novel derivative-free stochastic Runge-Kutta scheme, achieving the first provably higher convergence order than Euler-Maruyama. Numerical experiments on Gaussian toy models and CIFAR-10 confirm the predicted convergence orders.
We develop a convergence analysis for generative diffusion models that simultaneously accounts for the three principal sources of error in stochastic sampling: initialization error, score-matching error, and discretization of the reverse-time SDE. Our central tool is the notion of a general strong scheme, a broad class of discretization methods for the reverse dynamics defined via explicit, index-wise tolerances on their Itô-Taylor coefficients. This notion extends the classical strong-scheme framework of Kloeden and Platen to an iterate-wise formulation, which is strictly stronger and recovers their bound as a corollary. We prove a convergence theorem in the 2-Wasserstein distance that applies to this entire class of schemes at once, reducing the analysis of any concrete sampler to a finite verification checklist, and covers general forward processes with time-dependent, spatially linear drift and spatially independent diffusion coefficient, rather than a fixed variance-preserving, variance-exploding, or Ornstein--Uhlenbeck schedule. We instantiate this theorem for the Euler--Maruyama scheme, the exponential integrator, and, as our main application, a derivative-free stochastic Runge-Kutta scheme of strong order 1.5, yielding the first stochastic sampler for generative diffusion models with a provably higher convergence order than Euler--Maruyama. We further derive the resulting iteration complexity and an accompanying parameter-selection rule for the terminal time, score accuracy, and step size, and discuss the dissipative setting, in which the discretization and score-matching errors decouple from the terminal time. Numerical experiments on Gaussian toy models and the CIFAR-10 benchmark confirm the predicted convergence orders. Code available at: https://github.com/emanuelpfarr/SSGDM.