Please, Don't Forget the Difference and the Confidence Interval when Seeking for the State-of-the-Art Status
This addresses the problem of misleading comparisons in NLP research by promoting more rigorous evaluation methods, though it is incremental as it builds on existing statistical techniques.
The paper argues for using bootstrap confidence intervals instead of state-of-the-art status and statistical significance testing to compare NLP system performances, highlighting differences and assessing superiority, with case studies and a Python module provided.
This paper argues for the widest possible use of bootstrap confidence intervals for comparing NLP system performances instead of the state-of-the-art status (SOTA) and statistical significance testing. Their main benefits are to draw attention to the difference in performance between two systems and to help assessing the degree of superiority of one system over another. Two cases studies, one comparing several systems and the other based on a K-fold cross-validation procedure, illustrate these benefits. A python module for obtaining these confidence intervals as well as a second function implementing the Fisher-Pitman test for paired samples are freely available on PyPi.