CheckThat! at CLEF 2020: Enabling the Automatic Identification and Verification of Claims in Social Media
This work addresses the challenge of misinformation in social media for fact-checkers and researchers, but it is incremental as it builds on previous lab editions.
The paper tackles the problem of automatically identifying and verifying claims in social media by describing the third edition of the CheckThat! Lab, which includes tasks for predicting check-worthiness, verifying claims against fact-checked sources, retrieving evidence, and predicting veracity, with evaluation using metrics like mean average precision and F1 scores.
We describe the third edition of the CheckThat! Lab, which is part of the 2020 Cross-Language Evaluation Forum (CLEF). CheckThat! proposes four complementary tasks and a related task from previous lab editions, offered in English, Arabic, and Spanish. Task 1 asks to predict which tweets in a Twitter stream are worth fact-checking. Task 2 asks to determine whether a claim posted in a tweet can be verified using a set of previously fact-checked claims. Task 3 asks to retrieve text snippets from a given set of Web pages that would be useful for verifying a target tweet's claim. Task 4 asks to predict the veracity of a target tweet's claim using a set of Web pages and potentially useful snippets in them. Finally, the lab offers a fifth task that asks to predict the check-worthiness of the claims made in English political debates and speeches. CheckThat! features a full evaluation framework. The evaluation is carried out using mean average precision or precision at rank k for ranking tasks, and F1 for classification tasks.