What's Wrong with Your Synthetic Tabular Data? Using Explainable AI to Evaluate Generative Models
This provides a more transparent and diagnostic tool for researchers and practitioners working with generative models in tabular data, though it is incremental as it builds on existing XAI methods.
The paper tackles the challenge of evaluating synthetic tabular data by applying explainable AI (XAI) techniques to a binary detection classifier, revealing specific weaknesses like inconsistencies and unrealistic dependencies that standard metrics miss.
Evaluating synthetic tabular data is challenging, since they can differ from the real data in so many ways. There exist numerous metrics of synthetic data quality, ranging from statistical distances to predictive performance, often providing conflicting results. Moreover, they fail to explain or pinpoint the specific weaknesses in the synthetic data. To address this, we apply explainable AI (XAI) techniques to a binary detection classifier trained to distinguish real from synthetic data. While the classifier identifies distributional differences, XAI concepts such as feature importance and feature effects, analyzed through methods like permutation feature importance, partial dependence plots, Shapley values and counterfactual explanations, reveal why synthetic data are distinguishable, highlighting inconsistencies, unrealistic dependencies, or missing patterns. This interpretability increases transparency in synthetic data evaluation and provides deeper insights beyond conventional metrics, helping diagnose and improve synthetic data quality. We apply our approach to two tabular datasets and generative models, showing that it uncovers issues overlooked by standard evaluation techniques.