Comparing Knowledge Sources for Open-Domain Scientific Claim Verification
This work addresses the need for efficient fact-checking systems in real-world settings where evidence must be retrieved from large knowledge sources, though it is incremental as it compares existing methods without introducing new ones.
The paper tackled the problem of open-domain scientific claim verification by comparing different knowledge sources and retrieval methods, finding that PubMed performs better for biomedical claims and Wikipedia for everyday health concerns, with BM25 achieving higher precision and semantic search higher recall.
The increasing rate at which scientific knowledge is discovered and health claims shared online has highlighted the importance of developing efficient fact-checking systems for scientific claims. The usual setting for this task in the literature assumes that the documents containing the evidence for claims are already provided and annotated or contained in a limited corpus. This renders the systems unrealistic for real-world settings where knowledge sources with potentially millions of documents need to be queried to find relevant evidence. In this paper, we perform an array of experiments to test the performance of open-domain claim verification systems. We test the final verdict prediction of systems on four datasets of biomedical and health claims in different settings. While keeping the pipeline's evidence selection and verdict prediction parts constant, document retrieval is performed over three common knowledge sources (PubMed, Wikipedia, Google) and using two different information retrieval techniques. We show that PubMed works better with specialized biomedical claims, while Wikipedia is more suited for everyday health concerns. Likewise, BM25 excels in retrieval precision, while semantic search in recall of relevant evidence. We discuss the results, outline frequent retrieval patterns and challenges, and provide promising future directions.