Evaluating Differentially Private Generation of Domain-Specific Text
This addresses privacy barriers in high-stakes domains like healthcare and finance by providing systematic evaluation methods for synthetic data generation.
The authors tackled the problem of evaluating differentially private text generation for domain-specific applications by creating a unified benchmark that assesses utility and fidelity across five datasets, revealing significant degradation compared to real data under strict privacy constraints.
Generative AI offers transformative potential for high-stakes domains such as healthcare and finance, yet privacy and regulatory barriers hinder the use of real-world data. To address this, differentially private synthetic data generation has emerged as a promising alternative. In this work, we introduce a unified benchmark to systematically evaluate the utility and fidelity of text datasets generated under formal Differential Privacy (DP) guarantees. Our benchmark addresses key challenges in domain-specific benchmarking, including choice of representative data and realistic privacy budgets, accounting for pre-training and a variety of evaluation metrics. We assess state-of-the-art privacy-preserving generation methods across five domain-specific datasets, revealing significant utility and fidelity degradation compared to real data, especially under strict privacy constraints. These findings underscore the limitations of current approaches, outline the need for advanced privacy-preserving data sharing methods and set a precedent regarding their evaluation in realistic scenarios.