The Real Deal Behind the Artificial Appeal: Inferential Utility of Tabular Synthetic Data
This addresses a critical methodological challenge for researchers and practitioners using synthetic data in privacy-sensitive contexts, highlighting the need for new inference tools.
The paper tackles the problem of naive statistical inference from synthetic data generated by deep models, showing that it leads to unacceptably high false-positive rates (type 1 error) even with unbiased estimates, as demonstrated through a simulation study and case study.
Recent advances in generative models facilitate the creation of synthetic data to be made available for research in privacy-sensitive contexts. However, the analysis of synthetic data raises a unique set of methodological challenges. In this work, we highlight the importance of inferential utility and provide empirical evidence against naive inference from synthetic data, whereby synthetic data are treated as if they were actually observed. Before publishing synthetic data, it is essential to develop statistical inference tools for such data. By means of a simulation study, we show that the rate of false-positive findings (type 1 error) will be unacceptably high, even when the estimates are unbiased. Despite the use of a previously proposed correction factor, this problem persists for deep generative models, in part due to slower convergence of estimators and resulting underestimation of the true standard error. We further demonstrate our findings through a case study.