Universal Smoothed Score Functions for Generative Modeling
This work addresses generative modeling efficiency for researchers, offering incremental improvements in understanding and optimizing smoothed densities.
The paper tackles the problem of generative modeling by smoothing an unknown density using factorial kernels with Gaussian channels, deriving a universal parametrization for the smoothed density and analyzing its time complexity for learning and sampling. They report a Fréchet inception distance of 14.15 on CIFAR-10, achieved with a single noise level on fast-mixing MCMC chains.
We consider the problem of generative modeling based on smoothing an unknown density of interest in $\mathbb{R}^d$ using factorial kernels with $M$ independent Gaussian channels with equal noise levels introduced by Saremi and Srivastava (2022). First, we fully characterize the time complexity of learning the resulting smoothed density in $\mathbb{R}^{Md}$, called M-density, by deriving a universal form for its parametrization in which the score function is by construction permutation equivariant. Next, we study the time complexity of sampling an M-density by analyzing its condition number for Gaussian distributions. This spectral analysis gives a geometric insight on the "shape" of M-densities as one increases $M$. Finally, we present results on the sample quality in this class of generative models on the CIFAR-10 dataset where we report Fréchet inception distances (14.15), notably obtained with a single noise level on long-run fast-mixing MCMC chains.