Preserving Spectral Structure and Statistics in Diffusion Models
This addresses the problem of high computational cost and poor structural preservation in diffusion models for image generation, though it is incremental as it builds on existing diffusion frameworks.
The paper tackles the computational inefficiency and lack of structure in standard diffusion models by proposing a new method that uses an informative Gaussian prior in spectral space, resulting in significant reductions in computational complexity and improved visual diversity on datasets like CIFAR-10 and CelebA.
Standard diffusion models (DMs) rely on the total destruction of data into non-informative white noise, forcing the backward process to denoise from a fully unstructured noise state. While ensuring diversity, this results in a cumbersome and computationally intensive image generation task. We address this challenge by proposing new forward and backward process within a mathematically tractable spectral space. Unlike pixel-based DMs, our forward process converges towards an informative Gaussian prior N(mu_hat,Sigma_hat) rather than white noise. Our method, termed Preserving Spectral Structure and Statistics (PreSS) in diffusion models, guides spectral components toward this informative prior while ensuring that corresponding structural signals remain intact at terminal time. This provides a principled starting point for the backward process, enabling high-quality image reconstruction that builds upon preserved spectral structure while maintaining high generative diversity. Experimental results on CIFAR-10, CelebA and CelebA-HQ demonstrate significant reductions in computational complexity, improved visual diversity, less drift, and a smoother diffusion process compared to pixel-based DMs.