SynthEval: A Framework for Detailed Utility and Privacy Evaluation of Tabular Synthetic Data
This provides a versatile tool for researchers and practitioners in machine learning to benchmark and compare synthetic data models, though it is incremental as it builds on existing evaluation concepts.
The authors tackled the need for robust evaluation of synthetic tabular data by introducing SynthEval, an open-source framework that comprehensively assesses utility and privacy risks, treating categorical and numerical attributes equally without preprocessing assumptions.
With the growing demand for synthetic data to address contemporary issues in machine learning, such as data scarcity, data fairness, and data privacy, having robust tools for assessing the utility and potential privacy risks of such data becomes crucial. SynthEval, a novel open-source evaluation framework distinguishes itself from existing tools by treating categorical and numerical attributes with equal care, without assuming any special kind of preprocessing steps. This~makes it applicable to virtually any synthetic dataset of tabular records. Our tool leverages statistical and machine learning techniques to comprehensively evaluate synthetic data fidelity and privacy-preserving integrity. SynthEval integrates a wide selection of metrics that can be used independently or in highly customisable benchmark configurations, and can easily be extended with additional metrics. In this paper, we describe SynthEval and illustrate its versatility with examples. The framework facilitates better benchmarking and more consistent comparisons of model capabilities.