FEST: A Unified Framework for Evaluating Synthetic Tabular Data
This addresses the problem of evaluating synthetic data for researchers and practitioners, but it is incremental as it builds on existing metrics without introducing new methods.
The authors tackled the lack of a comprehensive evaluation framework for synthetic tabular data by proposing FEST, a systematic framework that integrates privacy and utility metrics, and they validated it on multiple datasets as an open-source library.
Synthetic data generation, leveraging generative machine learning techniques, offers a promising approach to mitigating privacy concerns associated with real-world data usage. Synthetic data closely resembles real-world data while maintaining strong privacy guarantees. However, a comprehensive assessment framework is still missing in the evaluation of synthetic data generation, especially when considering the balance between privacy preservation and data utility in synthetic data. This research bridges this gap by proposing FEST, a systematic framework for evaluating synthetic tabular data. FEST integrates diverse privacy metrics (attack-based and distance-based), along with similarity and machine learning utility metrics, to provide a holistic assessment. We develop FEST as an open-source Python-based library and validate it on multiple datasets, demonstrating its effectiveness in analyzing the privacy-utility trade-off of different synthetic data generation models. The source code of FEST is available on Github.