Exact Paired-Permutation Testing for Structured Test Statistics
This provides a more efficient and exact method for significance testing in NLP, addressing a bottleneck for practitioners who rely on approximations.
The paper tackles the problem of performing exact paired-permutation tests for structured test statistics in NLP, which previously relied on Monte Carlo approximations, and presents an efficient algorithm that runs in O(GN(log GN)(log N)) time, achieving a 10x speedup over Monte Carlo with 20000 samples on a common dataset.
Significance testing -- especially the paired-permutation test -- has played a vital role in developing NLP systems to provide confidence that the difference in performance between two systems (i.e., the test statistic) is not due to luck. However, practitioners rely on Monte Carlo approximation to perform this test due to a lack of a suitable exact algorithm. In this paper, we provide an efficient exact algorithm for the paired-permutation test for a family of structured test statistics. Our algorithm runs in $\mathcal{O}(GN (\log GN )(\log N ))$ time where $N$ is the dataset size and $G$ is the range of the test statistic. We found that our exact algorithm was $10$x faster than the Monte Carlo approximation with $20000$ samples on a common dataset.