Large Language Models Require Curated Context for Reliable Political Fact-Checking -- Even with Reasoning and Web Search
This addresses the problem of unreliable automated fact-checking for users of AI systems, highlighting an incremental improvement through curated context.
The study evaluated 15 large language models on over 6,000 political claims and found that standard models performed poorly, with reasoning and web search offering minimal to moderate improvements, while a curated RAG system increased macro F1 by 233% on average.
Large language models (LLMs) have raised hopes for automated end-to-end fact-checking, but prior studies report mixed results. As mainstream chatbots increasingly ship with reasoning capabilities and web search tools -- and millions of users already rely on them for verification -- rigorous evaluation is urgent. We evaluate 15 recent LLMs from OpenAI, Google, Meta, and DeepSeek on more than 6,000 claims fact-checked by PolitiFact, comparing standard models with reasoning- and web-search variants. Standard models perform poorly, reasoning offers minimal benefits, and web search provides only moderate gains, despite fact-checks being available on the web. In contrast, a curated RAG system using PolitiFact summaries improved macro F1 by 233% on average across model variants. These findings suggest that giving models access to curated high-quality context is a promising path for automated fact-checking.