Operator-Informed Score Matching for Markov Diffusion Models
This work addresses the challenge of training diffusion models more effectively by incorporating knowledge of Markov operators, offering potential improvements in sample generation and neural estimation.
This paper tackles the problem of training diffusion models by proposing operator-informed score matching, which leverages the spectral decomposition of Markov noising processes to estimate score functions for all marginal distributions using only data sample averages. The result is a method that provides a standalone approach for low-dimensional distributions and improves neural score estimators in high-dimensional settings.
Diffusion models are typically trained using score matching, a learning objective agnostic to the underlying noising process that guides the model. This paper argues that Markov noising processes enjoy an advantage over alternatives, as the Markov operators that govern the noising process are well-understood. Specifically, by leveraging the spectral decomposition of the infinitesimal generator of the Markov noising process, we obtain parametric estimates of the score functions simultaneously for all marginal distributions, using only sample averages with respect to the data distribution. The resulting operator-informed score matching provides both a standalone approach to sample generation for low-dimensional distributions, as well as a recipe for better informed neural score estimators in high-dimensional settings.