Revisiting Fake News Detection: Towards Temporality-aware Evaluation by Leveraging Engagement Earliness
This work addresses the challenge of realistic evaluation for fake news detection, which is crucial for social media platforms and users, though it is incremental in improving existing methods.
The paper tackles the problem of fake news detection by introducing a more realistic, temporality-aware evaluation scheme that reveals sharp performance drops in conventional methods, and proposes DAWN, which leverages engagement earliness to suppress noisy edges and outperforms existing methods in real-world environments.
Social graph-based fake news detection aims to identify news articles containing false information by utilizing social contexts, e.g., user information, tweets and comments. However, conventional methods are evaluated under less realistic scenarios, where the model has access to future knowledge on article-related and context-related data during training. In this work, we newly formalize a more realistic evaluation scheme that mimics real-world scenarios, where the data is temporality-aware and the detection model can only be trained on data collected up to a certain point in time. We show that the discriminative capabilities of conventional methods decrease sharply under this new setting, and further propose DAWN, a method more applicable to such scenarios. Our empirical findings indicate that later engagements (e.g., consuming or reposting news) contribute more to noisy edges that link real news-fake news pairs in the social graph. Motivated by this, we utilize feature representations of engagement earliness to guide an edge weight estimator to suppress the weights of such noisy edges, thereby enhancing the detection performance of DAWN. Through extensive experiments, we demonstrate that DAWN outperforms existing fake news detection methods under real-world environments. The source code is available at https://github.com/LeeJunmo/DAWN.