Stop Using the Wilcoxon Test: Myth, Misconception and Misuse in IR Research
For IR researchers, this work corrects a long-standing methodological misconception that has compromised the validity of statistical comparisons in system evaluations.
The paper argues that the Wilcoxon signed-rank test is routinely misapplied in IR evaluation, leading to inflated Type I error rates, and demonstrates through empirical analysis with TREC data that the t-test is a more appropriate alternative.
In benchmarking of Information Retrieval systems, the Wilcoxon signed-rank test is often treated as a safer alternative to the t-test. This belief is fueled by textbooks and recommendations that portray Wilcoxon as the proper non-parametric alternative because metric scores are not normally distributed. We argue that this narrative is misleading and harmful. A careful review of Statistics textbooks reveals inconsistencies and omissions in how the assumptions underlying these tests are presented, fostering confusion that has propagated into IR research. As a result, Wilcoxon has been routinely misapplied for decades, creating a false sense of safety against a threat that was never there to begin with, while introducing another one so severe that it virtually guarantees the test will break down and mislead researchers. Through a combination of systematic literature review, analysis and empirical demonstrations with TREC data, we show how and why the Wilcoxon test easily loses control of its Type I error rate in IR settings. We conclude that the continued use of Wilcoxon in IR evaluation is unjustified and that abandoning it would improve the methodological soundness of our field.