Benchmarking Synthetic Tabular Data: A Multi-Dimensional Evaluation Framework
This work addresses the need for consistent and reproducible benchmarking in synthetic data generation, which is crucial for researchers and practitioners in data-driven fields, though it is incremental as it builds on existing evaluation methods.
The authors tackled the challenge of evaluating synthetic tabular data quality by developing a multi-dimensional evaluation framework that quantifies distributional replication and privacy, resulting in a standardized benchmarking approach with interpretable metrics.
Evaluating the quality of synthetic data remains a key challenge for ensuring privacy and utility in data-driven research. In this work, we present an evaluation framework that quantifies how well synthetic data replicates original distributional properties while ensuring privacy. The proposed approach employs a holdout-based benchmarking strategy that facilitates quantitative assessment through low- and high-dimensional distribution comparisons, embedding-based similarity measures, and nearest-neighbor distance metrics. The framework supports various data types and structures, including sequential and contextual information, and enables interpretable quality diagnostics through a set of standardized metrics. These contributions aim to support reproducibility and methodological consistency in benchmarking of synthetic data generation techniques. The code of the framework is available at https://github.com/mostly-ai/mostlyai-qa.