Automated Claim Matching with Large Language Models: Empowering Fact-Checkers in the Fight Against Misinformation
This work addresses the problem of misinformation spread for fact-checkers and the public, offering an incremental improvement by automating a specific phase of fact-checking with LLMs.
The paper tackles the challenge of automating claim matching in fact-checking to combat misinformation by introducing FACT-GPT, a framework that uses LLMs to identify social media content supporting or contradicting debunked claims, with results showing fine-tuned LLMs rivaling larger pre-trained models in performance aligned with human annotations.
In today's digital era, the rapid spread of misinformation poses threats to public well-being and societal trust. As online misinformation proliferates, manual verification by fact checkers becomes increasingly challenging. We introduce FACT-GPT (Fact-checking Augmentation with Claim matching Task-oriented Generative Pre-trained Transformer), a framework designed to automate the claim matching phase of fact-checking using Large Language Models (LLMs). This framework identifies new social media content that either supports or contradicts claims previously debunked by fact-checkers. Our approach employs GPT-4 to generate a labeled dataset consisting of simulated social media posts. This data set serves as a training ground for fine-tuning more specialized LLMs. We evaluated FACT-GPT on an extensive dataset of social media content related to public health. The results indicate that our fine-tuned LLMs rival the performance of larger pre-trained LLMs in claim matching tasks, aligning closely with human annotations. This study achieves three key milestones: it provides an automated framework for enhanced fact-checking; demonstrates the potential of LLMs to complement human expertise; offers public resources, including datasets and models, to further research and applications in the fact-checking domain.