A Kernelised Stein Statistic for Assessing Implicit Generative Models
This addresses the need for reliable evaluation of synthetic data generators in machine learning, which is crucial for tasks like data augmentation and privacy-sensitive analysis, though it is an incremental improvement over existing testing methods.
The paper tackles the problem of assessing the quality of implicit generative models, which lack explicit probability distributions, by proposing a kernelised Stein discrepancy test that uses a non-parametric Stein operator estimated from synthetic samples. The method shows improved power performance compared to existing approaches in experiments on synthetic and real datasets.
Synthetic data generation has become a key ingredient for training machine learning procedures, addressing tasks such as data augmentation, analysing privacy-sensitive data, or visualising representative samples. Assessing the quality of such synthetic data generators hence has to be addressed. As (deep) generative models for synthetic data often do not admit explicit probability distributions, classical statistical procedures for assessing model goodness-of-fit may not be applicable. In this paper, we propose a principled procedure to assess the quality of a synthetic data generator. The procedure is a kernelised Stein discrepancy (KSD)-type test which is based on a non-parametric Stein operator for the synthetic data generator of interest. This operator is estimated from samples which are obtained from the synthetic data generator and hence can be applied even when the model is only implicit. In contrast to classical testing, the sample size from the synthetic data generator can be as large as desired, while the size of the observed data, which the generator aims to emulate is fixed. Experimental results on synthetic distributions and trained generative models on synthetic and real datasets illustrate that the method shows improved power performance compared to existing approaches.