Diffusion Models and the Manifold Hypothesis: Log-Domain Smoothing is Geometry Adaptive
This provides theoretical evidence for a key conjecture in understanding diffusion models, which is incremental but clarifies mechanisms for researchers.
The paper investigates whether diffusion models' success stems from adapting to low-dimensional data geometry, as per the manifold hypothesis, and finds that smoothing the score function leads to tangential smoothing along the data manifold, with controllable generalization.
Diffusion models have achieved state-of-the-art performance, demonstrating remarkable generalisation capabilities across diverse domains. However, the mechanisms underpinning these strong capabilities remain only partially understood. A leading conjecture, based on the manifold hypothesis, attributes this success to their ability to adapt to low-dimensional geometric structure within the data. This work provides evidence for this conjecture, focusing on how such phenomena could result from the formulation of the learning problem through score matching. We inspect the role of implicit regularisation by investigating the effect of smoothing minimisers of the empirical score matching objective. Our theoretical and empirical results confirm that smoothing the score function -- or equivalently, smoothing in the log-density domain -- produces smoothing tangential to the data manifold. In addition, we show that the manifold along which the diffusion model generalises can be controlled by choosing an appropriate smoothing.