Not All Claims are Created Equal: Choosing the Right Statistical Approach to Assess Hypotheses
This addresses a methodological gap for NLP researchers, offering incremental improvements in statistical rigor to reduce common fallacies and misinterpretations in the field.
The paper tackles the problem of narrow statistical practices in NLP hypothesis testing, which rely heavily on p-values with known issues, by contrasting alternative methods like Bayesian inference and providing guidelines and a tool called 'HyBayes' to improve assessment accuracy.
Empirical research in Natural Language Processing (NLP) has adopted a narrow set of principles for assessing hypotheses, relying mainly on p-value computation, which suffers from several known issues. While alternative proposals have been well-debated and adopted in other fields, they remain rarely discussed or used within the NLP community. We address this gap by contrasting various hypothesis assessment techniques, especially those not commonly used in the field (such as evaluations based on Bayesian inference). Since these statistical techniques differ in the hypotheses they can support, we argue that practitioners should first decide their target hypothesis before choosing an assessment method. This is crucial because common fallacies, misconceptions, and misinterpretation surrounding hypothesis assessment methods often stem from a discrepancy between what one would like to claim versus what the method used actually assesses. Our survey reveals that these issues are omnipresent in the NLP research community. As a step forward, we provide best practices and guidelines tailored to NLP research, as well as an easy-to-use package called 'HyBayes' for Bayesian assessment of hypotheses, complementing existing tools.