FIRE: Fact-checking with Iterative Retrieval and Verification
This addresses the challenge of cost-effective fact-checking for applications like large-scale operations, representing an incremental improvement over existing methods.
The paper tackles the problem of fact-checking long-form text by proposing FIRE, an agent-based framework that integrates iterative retrieval and verification, achieving slightly better performance while reducing LLM costs by 7.6 times and search costs by 16.5 times.
Fact-checking long-form text is challenging, and it is therefore common practice to break it down into multiple atomic claims. The typical approach to fact-checking these atomic claims involves retrieving a fixed number of pieces of evidence, followed by a verification step. However, this method is usually not cost-effective, as it underutilizes the verification model's internal knowledge of the claim and fails to replicate the iterative reasoning process in human search strategies. To address these limitations, we propose FIRE, a novel agent-based framework that integrates evidence retrieval and claim verification in an iterative manner. Specifically, FIRE employs a unified mechanism to decide whether to provide a final answer or generate a subsequent search query, based on its confidence in the current judgment. We compare FIRE with other strong fact-checking frameworks and find that it achieves slightly better performance while reducing large language model (LLM) costs by an average of 7.6 times and search costs by 16.5 times. These results indicate that FIRE holds promise for application in large-scale fact-checking operations. Our code is available at https://github.com/mbzuai-nlp/fire.git.