Generative Modelling with High-Order Langevin Dynamics
This work addresses efficiency and quality bottlenecks in generative modeling for image synthesis, representing a novel method rather than an incremental improvement.
The paper tackles the problem of slow mixing times in diffusion generative models by proposing high-order Langevin dynamics (HOLD), which simultaneously models position, velocity, and acceleration to improve generation quality and speed. The method achieves a state-of-the-art FID of 1.85 on CIFAR-10 and reduces mixing time by two orders of magnitude.
Diffusion generative modelling (DGM) based on stochastic differential equations (SDEs) with score matching has achieved unprecedented results in data generation. In this paper, we propose a novel fast high-quality generative modelling method based on high-order Langevin dynamics (HOLD) with score matching. This motive is proved by third-order Langevin dynamics. By augmenting the previous SDEs, e.g. variance exploding or variance preserving SDEs for single-data variable processes, HOLD can simultaneously model position, velocity, and acceleration, thereby improving the quality and speed of the data generation at the same time. HOLD is composed of one Ornstein-Uhlenbeck process and two Hamiltonians, which reduce the mixing time by two orders of magnitude. Empirical experiments for unconditional image generation on the public data set CIFAR-10 and CelebA-HQ show that the effect is significant in both Frechet inception distance (FID) and negative log-likelihood, and achieves the state-of-the-art FID of 1.85 on CIFAR-10.