Consistent Diffusion Meets Tweedie: Training Exact Ambient Diffusion Models with Noisy Data
This solves an open problem in ambient diffusion for researchers and practitioners, enabling training with corrupted data to mitigate copyright and privacy concerns from memorization.
The authors tackled the problem of training diffusion models using only noisy data, presenting the first framework that provably samples from the uncorrupted distribution, and demonstrated it by fine-tuning Stable Diffusion XL to reduce memorization while maintaining competitive performance.
Ambient diffusion is a recently proposed framework for training diffusion models using corrupted data. Both Ambient Diffusion and alternative SURE-based approaches for learning diffusion models from corrupted data resort to approximations which deteriorate performance. We present the first framework for training diffusion models that provably sample from the uncorrupted distribution given only noisy training data, solving an open problem in this space. Our key technical contribution is a method that uses a double application of Tweedie's formula and a consistency loss function that allows us to extend sampling at noise levels below the observed data noise. We also provide further evidence that diffusion models memorize from their training sets by identifying extremely corrupted images that are almost perfectly reconstructed, raising copyright and privacy concerns. Our method for training using corrupted samples can be used to mitigate this problem. We demonstrate this by fine-tuning Stable Diffusion XL to generate samples from a distribution using only noisy samples. Our framework reduces the amount of memorization of the fine-tuning dataset, while maintaining competitive performance.