Stochastic Forward-Backward Deconvolution: Training Diffusion Models with Finite Noisy Datasets
This addresses copyright concerns in generative AI by enabling training with minimal clean data, though it is incremental as it builds on existing diffusion models.
The paper tackles the problem of training diffusion models without memorizing copyrighted data by using noisy samples, showing that learning from noise alone is impractical due to sample inefficiency. They propose a method combining limited clean pretraining with Stochastic Forward-Backward Deconvolution, achieving FID scores of 6.31 on CIFAR-10 with 4% clean images and 3.58 with 10%.
Recent diffusion-based generative models achieve remarkable results by training on massive datasets, yet this practice raises concerns about memorization and copyright infringement. A proposed remedy is to train exclusively on noisy data with potential copyright issues, ensuring the model never observes original content. However, through the lens of deconvolution theory, we show that although it is theoretically feasible to learn the data distribution from noisy samples, the practical challenge of collecting sufficient samples makes successful learning nearly unattainable. To overcome this limitation, we propose to pretrain the model with a small fraction of clean data to guide the deconvolution process. Combined with our Stochastic Forward--Backward Deconvolution (SFBD) method, we attain FID 6.31 on CIFAR-10 with just 4% clean images (and 3.58 with 10%). We also provide theoretical guarantees that SFBD learns the true data distribution. These results underscore the value of limited clean pretraining, or pretraining on similar datasets. Empirical studies further validate and enrich our findings.