Informative Features for Model Comparison
This addresses the need for interpretable and efficient model comparison in machine learning, though it is incremental as it builds on existing relative goodness-of-fit tests.
The paper tackles the problem of measuring relative goodness of fit between two models by proposing two nonparametric, computationally efficient statistical tests that can identify specific data regions where one model outperforms the other. In a real-world GAN comparison, the new test matches state-of-the-art power while being ten times faster.
Given two candidate models, and a set of target observations, we address the problem of measuring the relative goodness of fit of the two models. We propose two new statistical tests which are nonparametric, computationally efficient (runtime complexity is linear in the sample size), and interpretable. As a unique advantage, our tests can produce a set of examples (informative features) indicating the regions in the data domain where one model fits significantly better than the other. In a real-world problem of comparing GAN models, the test power of our new test matches that of the state-of-the-art test of relative goodness of fit, while being one order of magnitude faster.