Analysis of Diffusion Models for Manifold Data
This work provides theoretical insights into diffusion models for manifold-structured data, which is incremental as it extends prior analysis to a more specific data model.
The authors analyzed the dynamics of generative diffusion models for data lying on manifolds, specifically computing transition times between different dynamical regimes for a mixture of lower-dimensional Gaussians embedded in higher-dimensional space.
We analyze the time reversed dynamics of generative diffusion models. If the exact empirical score function is used in a regime of large dimension and exponentially large number of samples, these models are known to undergo transitions between distinct dynamical regimes. We extend this analysis and compute the transitions for an analytically tractable manifold model where the statistical model for the data is a mixture of lower dimensional Gaussians embedded in higher dimensional space. We compute the so-called speciation and collapse transition times, as a function of the ratio of manifold-to-ambient space dimensions, and other characteristics of the data model. An important tool used in our analysis is the exact formula for the mutual information (or free energy) of Generalized Linear Models.