Free Hunch: Denoiser Covariance Estimation for Diffusion Models Without Extra Costs
This work addresses a bottleneck in training-free guided generation for diffusion models, offering a more efficient solution for researchers and practitioners in generative AI.
The paper tackles the problem of estimating clean data covariance from noisy observations in diffusion models without extra computational costs, achieving superior performance on linear inverse problems, particularly with fewer diffusion steps.
The covariance for clean data given a noisy observation is an important quantity in many training-free guided generation methods for diffusion models. Current methods require heavy test-time computation, altering the standard diffusion training process or denoiser architecture, or making heavy approximations. We propose a new framework that sidesteps these issues by using covariance information that is available for free from training data and the curvature of the generative trajectory, which is linked to the covariance through the second-order Tweedie's formula. We integrate these sources of information using (i) a novel method to transfer covariance estimates across noise levels and (ii) low-rank updates in a given noise level. We validate the method on linear inverse problems, where it outperforms recent baselines, especially with fewer diffusion steps.