Provable Separations between Memorization and Generalization in Diffusion Models
This addresses privacy and safety concerns in diffusion models for AI applications, though it is incremental as it builds on existing theoretical gaps.
The paper tackles the problem of memorization in diffusion models, where models reproduce training data instead of generating novel outputs, by developing a dual-separation result from statistical estimation and network approximation perspectives, showing that the ground-truth score function does not minimize empirical loss and requires larger networks, and proposes a pruning-based method to reduce memorization while maintaining quality.
Diffusion models have achieved remarkable success across diverse domains, but they remain vulnerable to memorization -- reproducing training data rather than generating novel outputs. This not only limits their creative potential but also raises concerns about privacy and safety. While empirical studies have explored mitigation strategies, theoretical understanding of memorization remains limited. We address this gap through developing a dual-separation result via two complementary perspectives: statistical estimation and network approximation. From the estimation side, we show that the ground-truth score function does not minimize the empirical denoising loss, creating a separation that drives memorization. From the approximation side, we prove that implementing the empirical score function requires network size to scale with sample size, spelling a separation compared to the more compact network representation of the ground-truth score function. Guided by these insights, we develop a pruning-based method that reduces memorization while maintaining generation quality in diffusion transformers.