RIGOURATE: Quantifying Scientific Exaggeration with Evidence-Aligned Claim Evaluation
This work addresses the issue of overstated claims in scientific papers, particularly in AI/ML conferences like ICLR and NeurIPS, by providing a tool for more transparent communication, though it is incremental as it builds on existing methods for evidence retrieval and claim evaluation.
The authors tackled the problem of scientific exaggeration by developing RIGOURATE, a multimodal framework that retrieves evidence from papers and assigns overstatement scores to claims, using a dataset of over 10K claim-evidence sets from ICLR and NeurIPS papers and achieving improved evidence retrieval and overstatement detection compared to baselines.
Scientific rigour tends to be sidelined in favour of bold statements, leading authors to overstate claims beyond what their results support. We present RIGOURATE, a two-stage multimodal framework that retrieves supporting evidence from a paper's body and assigns each claim an overstatement score. The framework consists of a dataset of over 10K claim-evidence sets from ICLR and NeurIPS papers, annotated using eight LLMs, with overstatement scores calibrated using peer-review comments and validated through human evaluation. It employes a fine-tuned reranker for evidence retrieval and a fine-tuned model to predict overstatement scores with justification. Compared to strong baselines, RIGOURATE enables improved evidence retrieval and overstatement detection. Overall, our work operationalises evidential proportionality and supports clearer, more transparent scientific communication.