QuanTemp: A real-world open-domain benchmark for fact-checking numerical claims
This addresses the challenge of fact-checking complex numerical claims in real-world scenarios, which is incremental as it builds on existing datasets by focusing on numerical aspects.
The authors tackled the problem of verifying real-world numerical claims by releasing QuanTemp, a diverse dataset with fine-grained metadata, and found that existing methods achieve a macro-F1 of 58.32, showing it is a challenging benchmark.
Automated fact checking has gained immense interest to tackle the growing misinformation in the digital era. Existing systems primarily focus on synthetic claims on Wikipedia, and noteworthy progress has also been made on real-world claims. In this work, we release QuanTemp, a diverse, multi-domain dataset focused exclusively on numerical claims, encompassing temporal, statistical and diverse aspects with fine-grained metadata and an evidence collection without leakage. This addresses the challenge of verifying real-world numerical claims, which are complex and often lack precise information, not addressed by existing works that mainly focus on synthetic claims. We evaluate and quantify the limitations of existing solutions for the task of verifying numerical claims. We also evaluate claim decomposition based methods, numerical understanding based models and our best baselines achieves a macro-F1 of 58.32. This demonstrates that QuanTemp serves as a challenging evaluation set for numerical claim verification.