Weakly Supervised Veracity Classification with LLM-Predicted Credibility Signals
This addresses misinformation detection for journalists and fact-checkers by reducing reliance on human annotations, though it is incremental as it builds on existing credibility signal concepts.
The paper tackles the problem of automating veracity classification of online content by introducing Pastel, a weakly supervised approach that uses large language models to extract credibility signals, achieving 86.7% of the performance of state-of-the-art supervised systems and outperforming zero-shot detection by 38.3%.
Credibility signals represent a wide range of heuristics typically used by journalists and fact-checkers to assess the veracity of online content. Automating the extraction of credibility signals presents significant challenges due to the necessity of training high-accuracy, signal-specific extractors, coupled with the lack of sufficiently large annotated datasets. This paper introduces Pastel (Prompted weAk Supervision wiTh crEdibility signaLs), a weakly supervised approach that leverages large language models (LLMs) to extract credibility signals from web content, and subsequently combines them to predict the veracity of content without relying on human supervision. We validate our approach using four article-level misinformation detection datasets, demonstrating that Pastel outperforms zero-shot veracity detection by 38.3% and achieves 86.7% of the performance of the state-of-the-art system trained with human supervision. Moreover, in cross-domain settings where training and testing datasets originate from different domains, Pastel significantly outperforms the state-of-the-art supervised model by 63%. We further study the association between credibility signals and veracity, and perform an ablation study showing the impact of each signal on model performance. Our findings reveal that 12 out of the 19 proposed signals exhibit strong associations with veracity across all datasets, while some signals show domain-specific strengths.