CSMeD: Bridging the Dataset Gap in Automated Citation Screening for Systematic Literature Reviews
This addresses a dataset gap for researchers developing machine learning tools to automate literature screening, though it is incremental as it builds on existing collections.
The paper tackles the lack of standardized datasets for comparing automated citation screening systems in systematic literature reviews by introducing CSMeD, a meta-dataset consolidating 325 SLRs from medicine and computer science, and CSMeD-FT for full-text screening, with experiments establishing baselines on these new datasets.
Systematic literature reviews (SLRs) play an essential role in summarising, synthesising and validating scientific evidence. In recent years, there has been a growing interest in using machine learning techniques to automate the identification of relevant studies for SLRs. However, the lack of standardised evaluation datasets makes comparing the performance of such automated literature screening systems difficult. In this paper, we analyse the citation screening evaluation datasets, revealing that many of the available datasets are either too small, suffer from data leakage or have limited applicability to systems treating automated literature screening as a classification task, as opposed to, for example, a retrieval or question-answering task. To address these challenges, we introduce CSMeD, a meta-dataset consolidating nine publicly released collections, providing unified access to 325 SLRs from the fields of medicine and computer science. CSMeD serves as a comprehensive resource for training and evaluating the performance of automated citation screening models. Additionally, we introduce CSMeD-FT, a new dataset designed explicitly for evaluating the full text publication screening task. To demonstrate the utility of CSMeD, we conduct experiments and establish baselines on new datasets.