Statistical Inference for Generative Model Comparison
This addresses the lack of statistical confidence in generative model evaluation for researchers and practitioners, though it is incremental as it builds on existing divergence measures.
The paper tackles the problem of evaluating generative models by developing a method for principled uncertainty quantification in model comparison, using KL divergence and Edgeworth expansions, and shows effective coverage rates and higher power than kernel-based methods on simulated and real datasets.
Generative models have achieved remarkable success across a range of applications, yet their evaluation still lacks principled uncertainty quantification. In this paper, we develop a method for comparing how close different generative models are to the underlying distribution of test samples. Particularly, our approach employs the Kullback-Leibler (KL) divergence to measure the distance between a generative model and the unknown test distribution, as KL requires no tuning parameters such as the kernels used by RKHS-based distances, and is the only $f$-divergence that admits a crucial cancellation to enable the uncertainty quantification. Furthermore, we extend our method to comparing conditional generative models and leverage Edgeworth expansions to address limited-data settings. On simulated datasets with known ground truth, we show that our approach realizes effective coverage rates, and has higher power compared to kernel-based methods. When applied to generative models on image and text datasets, our procedure yields conclusions consistent with benchmark metrics but with statistical confidence.