Evaluation of the Synthetic Electronic Health Records
This work addresses the challenge of comparing generative models for sensitive medical data, focusing on privacy and utility, but is incremental as it builds on existing evaluation methods.
The paper tackled the problem of evaluating synthetic electronic health records by proposing two new metrics, Similarity and Uniqueness, for sample-wise assessment, and demonstrated their effectiveness on Cystic Fibrosis patient data with state-of-the-art generative models.
Generative models have been found effective for data synthesis due to their ability to capture complex underlying data distributions. The quality of generated data from these models is commonly evaluated by visual inspection for image datasets or downstream analytical tasks for tabular datasets. These evaluation methods neither measure the implicit data distribution nor consider the data privacy issues, and it remains an open question of how to compare and rank different generative models. Medical data can be sensitive, so it is of great importance to draw privacy concerns of patients while maintaining the data utility of the synthetic dataset. Beyond the utility evaluation, this work outlines two metrics called Similarity and Uniqueness for sample-wise assessment of synthetic datasets. We demonstrate the proposed notions with several state-of-the-art generative models to synthesise Cystic Fibrosis (CF) patients' electronic health records (EHRs), observing that the proposed metrics are suitable for synthetic data evaluation and generative model comparison.