Capacity Control is an Effective Memorization Mitigation Mechanism in Text-Conditional Diffusion Models
This addresses memorization issues in diffusion models for medical imaging, but it is incremental as it builds on existing fine-tuning paradigms.
The authors tackled memorization in text-conditional diffusion models by showing that controlling model capacity through Parameter-Efficient Fine-Tuning (PEFT) reduces memorization compared to full fine-tuning, with experiments on the MIMIC dataset demonstrating improved generation quality.
In this work, we present compelling evidence that controlling model capacity during fine-tuning can effectively mitigate memorization in diffusion models. Specifically, we demonstrate that adopting Parameter-Efficient Fine-Tuning (PEFT) within the pre-train fine-tune paradigm significantly reduces memorization compared to traditional full fine-tuning approaches. Our experiments utilize the MIMIC dataset, which comprises image-text pairs of chest X-rays and their corresponding reports. The results, evaluated through a range of memorization and generation quality metrics, indicate that PEFT not only diminishes memorization but also enhances downstream generation quality. Additionally, PEFT methods can be seamlessly combined with existing memorization mitigation techniques for further improvement. The code for our experiments is available at: https://github.com/Raman1121/Diffusion_Memorization_HPO