B-tests: Low Variance Kernel Two-Sample Tests
This work addresses the need for efficient and powerful statistical tests for comparing distributions, which is incremental but offers practical improvements for machine learning and data analysis.
The authors tackled the problem of improving kernel two-sample tests by introducing B-tests, which balance test power and computational efficiency, achieving higher power than linear-time tests and lower computation than quadratic-time tests while providing an asymptotically Normal null distribution.
A family of maximum mean discrepancy (MMD) kernel two-sample tests is introduced. Members of the test family are called Block-tests or B-tests, since the test statistic is an average over MMDs computed on subsets of the samples. The choice of block size allows control over the tradeoff between test power and computation time. In this respect, the $B$-test family combines favorable properties of previously proposed MMD two-sample tests: B-tests are more powerful than a linear time test where blocks are just pairs of samples, yet they are more computationally efficient than a quadratic time test where a single large block incorporating all the samples is used to compute a U-statistic. A further important advantage of the B-tests is their asymptotically Normal null distribution: this is by contrast with the U-statistic, which is degenerate under the null hypothesis, and for which estimates of the null distribution are computationally demanding. Recent results on kernel selection for hypothesis testing transfer seamlessly to the B-tests, yielding a means to optimize test power via kernel choice.