AFaCTA: Assisting the Annotation of Factual Claim Detection with Reliable LLM Annotators
This addresses scalability and generalizability issues in misinformation detection for fact-checking systems, though it is incremental as it builds on existing annotation methods with LLM assistance.
The paper tackles the problem of factual claim detection in automated fact-checking by proposing a unifying definition of claims based on verifiability and introducing AFaCTA, a framework that uses large language models to assist annotation, resulting in the creation of the PoliClaim dataset for political speech.
With the rise of generative AI, automated fact-checking methods to combat misinformation are becoming more and more important. However, factual claim detection, the first step in a fact-checking pipeline, suffers from two key issues that limit its scalability and generalizability: (1) inconsistency in definitions of the task and what a claim is, and (2) the high cost of manual annotation. To address (1), we review the definitions in related work and propose a unifying definition of factual claims that focuses on verifiability. To address (2), we introduce AFaCTA (Automatic Factual Claim deTection Annotator), a novel framework that assists in the annotation of factual claims with the help of large language models (LLMs). AFaCTA calibrates its annotation confidence with consistency along three predefined reasoning paths. Extensive evaluation and experiments in the domain of political speech reveal that AFaCTA can efficiently assist experts in annotating factual claims and training high-quality classifiers, and can work with or without expert supervision. Our analyses also result in PoliClaim, a comprehensive claim detection dataset spanning diverse political topics.