Resolving Memorization in Empirical Diffusion Model for Manifold Data in High-Dimensional Spaces
This solves a memorization issue in diffusion models for manifold data, enabling generation of novel samples without extra training, which is incremental as it builds on existing empirical diffusion methods.
The paper tackles the memorization problem in empirical diffusion models, where models reproduce existing data points, by introducing an inertia update at the end of the simulation, which requires no additional training. The result is a Wasserstein-1 distance bound of O(n^{-2/(d+4)}) between the sample distribution and the true data distribution on a manifold, independent of ambient space dimension, producing new and diverse samples.
Diffusion models are popular tools for generating new data samples, using a forward process that adds noise to data and a reverse process to denoise and produce samples. However, when the data distribution consists of n points, empirical diffusion models tend to reproduce existing data points, a phenomenon known as the memorization effect. Current literature often addresses this with complex machine learning techniques. This work shows that the memorization issue can be solved simply by applying an inertia update at the end of the empirical diffusion simulation. Our inertial diffusion model requires only the empirical score function and no additional training. We demonstrate that the distribution of samples from this model approximates the true data distribution on a $C^2$ manifold of dimension $d$, within a Wasserstein-1 distance of order $O(n^{-\frac{2}{d+4}})$. This bound significantly shrinks the Wasserstein distance between the population and empirical distributions, confirming that the inertial diffusion model produces new and diverse samples. Remarkably, this estimate is independent of the ambient space dimension, as no further training is needed. Our analysis shows that the inertial diffusion samples resemble Gaussian kernel density estimations on the manifold, revealing a novel connection between diffusion models and manifold learning.