SciFact-Open: Towards open-domain scientific claim verification
This work addresses the challenge of deploying claim verification systems in practice for researchers and developers, but it is incremental as it focuses on dataset creation and evaluation rather than a new method.
The authors tackled the problem of evaluating scientific claim verification systems in a realistic open-domain setting by creating SciFact-Open, a test collection with 500K research abstracts, and found that existing systems struggle to generalize, with performance drops of at least 15 F1.
While research on scientific claim verification has led to the development of powerful systems that appear to approach human performance, these approaches have yet to be tested in a realistic setting against large corpora of scientific literature. Moving to this open-domain evaluation setting, however, poses unique challenges; in particular, it is infeasible to exhaustively annotate all evidence documents. In this work, we present SciFact-Open, a new test collection designed to evaluate the performance of scientific claim verification systems on a corpus of 500K research abstracts. Drawing upon pooling techniques from information retrieval, we collect evidence for scientific claims by pooling and annotating the top predictions of four state-of-the-art scientific claim verification models. We find that systems developed on smaller corpora struggle to generalize to SciFact-Open, exhibiting performance drops of at least 15 F1. In addition, analysis of the evidence in SciFact-Open reveals interesting phenomena likely to appear when claim verification systems are deployed in practice, e.g., cases where the evidence supports only a special case of the claim. Our dataset is available at https://github.com/dwadden/scifact-open.