Towards Effective Extraction and Evaluation of Factual Claims
This addresses the problem of unreliable fact-checking due to poor claim extraction for researchers and practitioners, though it is incremental as it builds on existing methods with a new evaluation framework.
The paper tackles the lack of a standardized evaluation framework for claim extraction in LLM-generated content fact-checking, proposing a new framework and Claimify method that outperforms existing approaches by handling ambiguity and extracting high-confidence claims.
A common strategy for fact-checking long-form content generated by Large Language Models (LLMs) is extracting simple claims that can be verified independently. Since inaccurate or incomplete claims compromise fact-checking results, ensuring claim quality is critical. However, the lack of a standardized evaluation framework impedes assessment and comparison of claim extraction methods. To address this gap, we propose a framework for evaluating claim extraction in the context of fact-checking along with automated, scalable, and replicable methods for applying this framework, including novel approaches for measuring coverage and decontextualization. We also introduce Claimify, an LLM-based claim extraction method, and demonstrate that it outperforms existing methods under our evaluation framework. A key feature of Claimify is its ability to handle ambiguity and extract claims only when there is high confidence in the correct interpretation of the source text.