On the Inherent Privacy Properties of Discrete Denoising Diffusion Models
This work addresses privacy concerns for data scientists and practitioners using diffusion models to generate synthetic datasets, offering a foundational theoretical framework for understanding privacy guarantees, though it is incremental as it builds on existing empirical evaluations.
The paper tackles the lack of mathematical characterization of privacy in discrete diffusion models for synthetic data generation, providing theoretical bounds on per-instance differential privacy that show privacy leakage increases from (ε, O(1/s²ε)) to (ε, O(1/sε)) during the diffusion process and decays faster with higher diffusion coefficients, with empirical validation on synthetic and real-world datasets.
Privacy concerns have led to a surge in the creation of synthetic datasets, with diffusion models emerging as a promising avenue. Although prior studies have performed empirical evaluations on these models, there has been a gap in providing a mathematical characterization of their privacy-preserving capabilities. To address this, we present the pioneering theoretical exploration of the privacy preservation inherent in discrete diffusion models (DDMs) for discrete dataset generation. Focusing on per-instance differential privacy (pDP), our framework elucidates the potential privacy leakage for each data point in a given training dataset, offering insights into how the privacy loss of each point correlates with the dataset's distribution. Our bounds also show that training with $s$-sized data points leads to a surge in privacy leakage from $(ε, O(\frac{1}{s^2ε}))$-pDP to $(ε, O(\frac{1}{sε}))$-pDP of the DDM during the transition from the pure noise to the synthetic clean data phase, and a faster decay in diffusion coefficients amplifies the privacy guarantee. Finally, we empirically verify our theoretical findings on both synthetic and real-world datasets.