HerO at AVeriTeC: The Herd of Open Large Language Models for Verifying Real-World Claims
This work addresses the problem of verifying real-world claims for fact-checking systems, though it is incremental as it applies existing methods to a new task.
The paper tackled automated fact-checking by using only publicly available large language models for evidence retrieval, question generation, and veracity prediction, achieving second place on the AVeriTeC leaderboard with a score of 0.57.
To tackle the AVeriTeC shared task hosted by the FEVER-24, we introduce a system that only employs publicly available large language models (LLMs) for each step of automated fact-checking, dubbed the Herd of Open LLMs for verifying real-world claims (HerO). For evidence retrieval, a language model is used to enhance a query by generating hypothetical fact-checking documents. We prompt pretrained and fine-tuned LLMs for question generation and veracity prediction by crafting prompts with retrieved in-context samples. HerO achieved 2nd place on the leaderboard with the AVeriTeC score of 0.57, suggesting the potential of open LLMs for verifying real-world claims. For future research, we make our code publicly available at https://github.com/ssu-humane/HerO.