Claim Extraction for Fact-Checking: Data, Models, and Automated Metrics
This work addresses the scattered research in claim extraction by providing a unified dataset and evaluation framework, which is incremental but useful for researchers in NLP and fact-checking.
The paper tackles the problem of claim extraction for fact-checking by comparing various text generation methods and introduces the FEVERFact dataset with 17K claims from Wikipedia sentences. It proposes an evaluation framework with automated metrics that closely approximate human grading, achieving consistent model rankings on the hardest metric.
In this paper, we explore the problem of Claim Extraction using one-to-many text generation methods, comparing LLMs, small summarization models finetuned for the task, and a previous NER-centric baseline QACG. As the current publications on Claim Extraction, Fact Extraction, Claim Generation and Check-worthy Claim Detection are quite scattered in their means and terminology, we compile their common objectives, releasing the FEVERFact dataset, with 17K atomic factual claims extracted from 4K contextualised Wikipedia sentences, adapted from the original FEVER. We compile the known objectives into an Evaluation framework of: Atomicity, Fluency, Decontextualization, Faithfulness checked for each generated claim separately, and Focus and Coverage measured against the full set of predicted claims for a single input. For each metric, we implement a scale using a reduction to an already-explored NLP task. We validate our metrics against human grading of generic claims, to see that the model ranking on $F_{fact}$, our hardest metric, did not change and the evaluation framework approximates human grading very closely in terms of $F_1$ and RMSE.