Fact or Fiction? Can LLMs be Reliable Annotators for Political Truths?
It addresses the challenge of political misinformation for democratic processes by providing a scalable solution to manual fact-checking limitations.
This study tackled the problem of detecting political factuality in news articles by using large language models (LLMs) as annotators, resulting in a scalable and robust alternative to traditional fact-checking that enhances transparency and public trust in media.
Political misinformation poses significant challenges to democratic processes, shaping public opinion and trust in media. Manual fact-checking methods face issues of scalability and annotator bias, while machine learning models require large, costly labelled datasets. This study investigates the use of state-of-the-art large language models (LLMs) as reliable annotators for detecting political factuality in news articles. Using open-source LLMs, we create a politically diverse dataset, labelled for bias through LLM-generated annotations. These annotations are validated by human experts and further evaluated by LLM-based judges to assess the accuracy and reliability of the annotations. Our approach offers a scalable and robust alternative to traditional fact-checking, enhancing transparency and public trust in media.