What's the score? Automated Denoising Score Matching for Nonlinear Diffusions
This work addresses a bottleneck in generative modeling for researchers and practitioners by extending diffusion methods to nonlinear processes, though it appears incremental as it builds on existing score matching frameworks.
The authors tackled the limitation of diffusion-based generative modeling to linear processes with Gaussian priors by introducing local-DSM, a tractable denoising score matching method for nonlinear diffusions, enabling training on non-Gaussian priors and challenging datasets like CIFAR10.
Reversing a diffusion process by learning its score forms the heart of diffusion-based generative modeling and for estimating properties of scientific systems. The diffusion processes that are tractable center on linear processes with a Gaussian stationary distribution. This limits the kinds of models that can be built to those that target a Gaussian prior or more generally limits the kinds of problems that can be generically solved to those that have conditionally linear score functions. In this work, we introduce a family of tractable denoising score matching objectives, called local-DSM, built using local increments of the diffusion process. We show how local-DSM melded with Taylor expansions enables automated training and score estimation with nonlinear diffusion processes. To demonstrate these ideas, we use automated-DSM to train generative models using non-Gaussian priors on challenging low dimensional distributions and the CIFAR10 image dataset. Additionally, we use the automated-DSM to learn the scores for nonlinear processes studied in statistical physics.