A Revisit of Fake News Dataset with Augmented Fact-checking by ChatGPT
This work addresses bias in fake news detection datasets for researchers and practitioners, but it is incremental as it builds on existing datasets with LLM augmentation.
The paper tackles the problem of bias in existing fake news detection datasets by augmenting human-verified data with fact-checking from ChatGPT, creating the ChatGPT-FC dataset. It quantitatively analyzes differences between human and LLM assessments across credibility, timeliness, and political framing, finding that LLMs can serve as a preliminary screening method to mitigate human biases.
The proliferation of fake news has emerged as a critical issue in recent years, requiring significant efforts to detect it. However, the existing fake news detection datasets are sourced from human journalists, which are likely to have inherent bias limitations due to the highly subjective nature of this task. In this paper, we revisit the existing fake news dataset verified by human journalists with augmented fact-checking by large language models (ChatGPT), and we name the augmented fake news dataset ChatGPT-FC. We quantitatively analyze the distinctions and resemblances between human journalists and LLM in assessing news subject credibility, news creator credibility, time-sensitive, and political framing. Our findings highlight LLM's potential to serve as a preliminary screening method, offering a promising avenue to mitigate the inherent biases of human journalists and enhance fake news detection.