Asymptotic Learning Curves for Diffusion Models with Random Features Score and Manifold Data
This provides foundational insights into how diffusion models learn from structured data, with implications for efficiency in high-dimensional settings.
The paper tackles the theoretical analysis of denoising score matching in diffusion models when data lies on a low-dimensional manifold, using random feature neural networks, and finds that sample complexity scales linearly with intrinsic dimension for linear manifolds but diminishes for non-linear ones.
We study the theoretical behavior of denoising score matching--the learning task associated to diffusion models--when the data distribution is supported on a low-dimensional manifold and the score is parameterized using a random feature neural network. We derive asymptotically exact expressions for the test, train, and score errors in the high-dimensional limit. Our analysis reveals that, for linear manifolds the sample complexity required to learn the score function scales linearly with the intrinsic dimension of the manifold, rather than with the ambient dimension. Perhaps surprisingly, the benefits of low-dimensional structure starts to diminish once we have a non-linear manifold. These results indicate that diffusion models can benefit from structured data; however, the dependence on the specific type of structure is subtle and intricate.