Are Large Language Models Good Fact Checkers: A Preliminary Study
This is an incremental study assessing LLMs for fact-checking, relevant for researchers and practitioners in natural language processing and misinformation detection.
The study evaluated large language models (LLMs) for fact-checking tasks, finding they achieve competitive performance compared to smaller models in most scenarios but struggle with Chinese verification and pipeline handling due to language inconsistencies and hallucinations.
Recently, Large Language Models (LLMs) have drawn significant attention due to their outstanding reasoning capabilities and extensive knowledge repository, positioning them as superior in handling various natural language processing tasks compared to other language models. In this paper, we present a preliminary investigation into the potential of LLMs in fact-checking. This study aims to comprehensively evaluate various LLMs in tackling specific fact-checking subtasks, systematically evaluating their capabilities, and conducting a comparative analysis of their performance against pre-trained and state-of-the-art low-parameter models. Experiments demonstrate that LLMs achieve competitive performance compared to other small models in most scenarios. However, they encounter challenges in effectively handling Chinese fact verification and the entirety of the fact-checking pipeline due to language inconsistencies and hallucinations. These findings underscore the need for further exploration and research to enhance the proficiency of LLMs as reliable fact-checkers, unveiling the potential capability of LLMs and the possible challenges in fact-checking tasks.