Does Generation Require Memorization? Creative Diffusion Models using Ambient Diffusion
This addresses the memorization issue in generative AI for applications requiring creative outputs, representing a notable advance over prior methods that degraded quality.
The paper tackles the problem of diffusion models memorizing training data, especially with small datasets, which compromises creativity. It proposes a method using noisy data at large noise scales, significantly reducing memorization without sacrificing image quality across various settings.
There is strong empirical evidence that the state-of-the-art diffusion modeling paradigm leads to models that memorize the training set, especially when the training set is small. Prior methods to mitigate the memorization problem often lead to a decrease in image quality. Is it possible to obtain strong and creative generative models, i.e., models that achieve high generation quality and low memorization? Despite the current pessimistic landscape of results, we make significant progress in pushing the trade-off between fidelity and memorization. We first provide theoretical evidence that memorization in diffusion models is only necessary for denoising problems at low noise scales (usually used in generating high-frequency details). Using this theoretical insight, we propose a simple, principled method to train the diffusion models using noisy data at large noise scales. We show that our method significantly reduces memorization without decreasing the image quality, for both text-conditional and unconditional models and for a variety of data availability settings.