CrowdChecked: Detecting Previously Fact-Checked Claims in Social Media
This work addresses the credibility gap in automated fact-checking systems for social media users by leveraging crowd-sourced data to enhance detection of repeated claims.
The paper tackled the problem of detecting previously fact-checked claims in social media by mining a large-scale dataset of 330,000 tweet-article pairs and proposing an end-to-end framework using modified self-adaptive training, resulting in a two-point absolute improvement over the state of the art on the CLEF'21 CheckThat! test set.
While there has been substantial progress in developing systems to automate fact-checking, they still lack credibility in the eyes of the users. Thus, an interesting approach has emerged: to perform automatic fact-checking by verifying whether an input claim has been previously fact-checked by professional fact-checkers and to return back an article that explains their decision. This is a sensible approach as people trust manual fact-checking, and as many claims are repeated multiple times. Yet, a major issue when building such systems is the small number of known tweet--verifying article pairs available for training. Here, we aim to bridge this gap by making use of crowd fact-checking, i.e., mining claims in social media for which users have responded with a link to a fact-checking article. In particular, we mine a large-scale collection of 330,000 tweets paired with a corresponding fact-checking article. We further propose an end-to-end framework to learn from this noisy data based on modified self-adaptive training, in a distant supervision scenario. Our experiments on the CLEF'21 CheckThat! test set show improvements over the state of the art by two points absolute. Our code and datasets are available at https://github.com/mhardalov/crowdchecked-claims