SynthTextEval: Synthetic Text Data Generation and Evaluation for High-Stakes Domains
This addresses the need for standardized evaluation to enable privacy-preserving AI development in critical domains, but it is incremental as it consolidates existing metrics into a toolkit.
The authors tackled the problem of evaluating synthetic text data for high-stakes domains by developing SynthTextEval, a toolkit that assesses utility, fairness, privacy, distributional differences, and expert feedback, demonstrating its effectiveness on healthcare and law datasets.
We present SynthTextEval, a toolkit for conducting comprehensive evaluations of synthetic text. The fluency of large language model (LLM) outputs has made synthetic text potentially viable for numerous applications, such as reducing the risks of privacy violations in the development and deployment of AI systems in high-stakes domains. Realizing this potential, however, requires principled consistent evaluations of synthetic data across multiple dimensions: its utility in downstream systems, the fairness of these systems, the risk of privacy leakage, general distributional differences from the source text, and qualitative feedback from domain experts. SynthTextEval allows users to conduct evaluations along all of these dimensions over synthetic data that they upload or generate using the toolkit's generation module. While our toolkit can be run over any data, we highlight its functionality and effectiveness over datasets from two high-stakes domains: healthcare and law. By consolidating and standardizing evaluation metrics, we aim to improve the viability of synthetic text, and in-turn, privacy-preservation in AI development.