A Test of Relative Similarity For Model Selection in Generative Models
This addresses the problem of model selection for researchers using generative models, but it is incremental as it builds on existing maximum mean discrepancy methods.
The paper tackles the challenge of model selection in generative models when likelihoods are inaccessible by introducing a statistical test of relative similarity to determine which model generates samples closer to a real-world reference dataset. The result is a low-variance test that provides a meaningful ranking of model performance in experiments on deep generative models.
Probabilistic generative models provide a powerful framework for representing data that avoids the expense of manual annotation typically needed by discriminative approaches. Model selection in this generative setting can be challenging, however, particularly when likelihoods are not easily accessible. To address this issue, we introduce a statistical test of relative similarity, which is used to determine which of two models generates samples that are significantly closer to a real-world reference dataset of interest. We use as our test statistic the difference in maximum mean discrepancies (MMDs) between the reference dataset and each model dataset, and derive a powerful, low-variance test based on the joint asymptotic distribution of the MMDs between each reference-model pair. In experiments on deep generative models, including the variational auto-encoder and generative moment matching network, the tests provide a meaningful ranking of model performance as a function of parameter and training settings.