Fact or Fiction: Verifying Scientific Claims
This addresses the challenge of fact-checking scientific claims for researchers and the public, though it is incremental as it builds on existing verification methods.
The paper tackles the problem of verifying scientific claims by introducing a new task to select abstracts with evidence that supports or refutes claims, and constructs SciFact, a dataset of 1.4K claims with annotated evidence. They develop baseline models showing that domain adaptation improves performance, and demonstrate application to COVID-19 claims using the CORD-19 corpus.
We introduce scientific claim verification, a new task to select abstracts from the research literature containing evidence that SUPPORTS or REFUTES a given scientific claim, and to identify rationales justifying each decision. To study this task, we construct SciFact, a dataset of 1.4K expert-written scientific claims paired with evidence-containing abstracts annotated with labels and rationales. We develop baseline models for SciFact, and demonstrate that simple domain adaptation techniques substantially improve performance compared to models trained on Wikipedia or political news. We show that our system is able to verify claims related to COVID-19 by identifying evidence from the CORD-19 corpus. Our experiments indicate that SciFact will provide a challenging testbed for the development of new systems designed to retrieve and reason over corpora containing specialized domain knowledge. Data and code for this new task are publicly available at https://github.com/allenai/scifact. A leaderboard and COVID-19 fact-checking demo are available at https://scifact.apps.allenai.org.