Generating Natural Language Queries for More Effective Systematic Review Screening Prioritisation
This work addresses the inefficiency in medical systematic review screening by providing a practical solution that does not depend on ex post facto information, though it is incremental as it builds on existing BERT-based ranking methods.
The paper tackled the problem of screening prioritization in medical systematic reviews by exploring alternative query sources, such as Boolean queries and generative language models, to rank documents effectively without relying on the final review title. The best approach achieved similar effectiveness to using the final title, making it viable for practical use during screening.
Screening prioritisation in medical systematic reviews aims to rank the set of documents retrieved by complex Boolean queries. Prioritising the most important documents ensures that subsequent review steps can be carried out more efficiently and effectively. The current state of the art uses the final title of the review as a query to rank the documents using BERT-based neural rankers. However, the final title is only formulated at the end of the review process, which makes this approach impractical as it relies on ex post facto information. At the time of screening, only a rough working title is available, with which the BERT-based ranker performs significantly worse than with the final title. In this paper, we explore alternative sources of queries for prioritising screening, such as the Boolean query used to retrieve the documents to be screened and queries generated by instruction-based generative large-scale language models such as ChatGPT and Alpaca. Our best approach is not only viable based on the information available at the time of screening, but also has similar effectiveness to the final title.