DP-RDM: Adapting Diffusion Models to Private Domains Without Fine-Tuning
This addresses privacy concerns in generative AI for domains like healthcare or personal data by enabling adaptation to private domains without fine-tuning, though it is incremental as it builds on existing retrieval-augmented and DP methods.
The paper tackles the problem of sample-level memorization in text-to-image diffusion models by developing a differentially private retrieval-augmented generation algorithm that generates high-quality images with provable privacy guarantees, achieving a 3.5 point improvement in FID compared to public-only retrieval for up to 10,000 queries with a privacy budget of ε=10 on MS-COCO.
Text-to-image diffusion models have been shown to suffer from sample-level memorization, possibly reproducing near-perfect replica of images that they are trained on, which may be undesirable. To remedy this issue, we develop the first differentially private (DP) retrieval-augmented generation algorithm that is capable of generating high-quality image samples while providing provable privacy guarantees. Specifically, we assume access to a text-to-image diffusion model trained on a small amount of public data, and design a DP retrieval mechanism to augment the text prompt with samples retrieved from a private retrieval dataset. Our \emph{differentially private retrieval-augmented diffusion model} (DP-RDM) requires no fine-tuning on the retrieval dataset to adapt to another domain, and can use state-of-the-art generative models to generate high-quality image samples while satisfying rigorous DP guarantees. For instance, when evaluated on MS-COCO, our DP-RDM can generate samples with a privacy budget of $ε=10$, while providing a $3.5$ point improvement in FID compared to public-only retrieval for up to $10,000$ queries.