Evaluating quality in synthetic data generation for large tabular health datasets
For researchers and practitioners in health data privacy, this provides a comparative evaluation framework, though it is incremental as it applies existing methods to new datasets.
This study evaluates seven synthetic data generation models on large health datasets, proposing a methodology for assessing fidelity of joint distributions. The analysis reveals challenges in adhering to medical domain constraints.
There is no consensus in the field of synthetic data on concise metrics for quality evaluations or benchmarks on large health datasets, such as historical epidemiological data. This study presents an evaluation of seven recent models from major machine learning families. The models were evaluated using four different datasets, each with a distinct scale. To ensure a fair comparison, we systematically tuned the hyperparameters of each model for each dataset. We propose a methodology for evaluating the fidelity of synthesized joint distributions, aligning metrics with visualization on a single plot. This method is applicable to any dataset and is complemented by a domain-specific analysis of the German Cancer Registries' epidemiological dataset. The analysis reveals the challenges models face in strictly adhering to the medical domain. We hope this approach will serve as a foundational framework for guiding the selection of synthesizers and remain accessible to all stakeholders involved in releasing synthetic datasets.