Cutting Through the Clutter: The Potential of LLMs for Efficient Filtration in Systematic Literature Reviews
This work addresses the problem of inefficient literature review filtering for researchers, representing an incremental improvement through a novel tool.
The paper tackles the labor-intensive process of systematic literature reviews by evaluating Large Language Models (LLMs) to enhance efficiency and accuracy, showing that LLMs can reduce filtering time from weeks to minutes and achieve recall rates over 98.8%.
Systematic literature reviews (SLRs) are essential but labor-intensive due to high publication volumes and inefficient keyword-based filtering. To streamline this process, we evaluate Large Language Models (LLMs) for enhancing efficiency and accuracy in corpus filtration while minimizing manual effort. Our open-source tool LLMSurver presents a visual interface to utilize LLMs for literature filtration, evaluate the results, and refine queries in an interactive way. We assess the real-world performance of our approach in filtering over 8.3k articles during a recent survey construction, comparing results with human efforts. The findings show that recent LLM models can reduce filtering time from weeks to minutes. A consensus scheme ensures recall rates >98.8%, surpassing typical human error thresholds and improving selection accuracy. This work advances literature review methodologies and highlights the potential of responsible human-AI collaboration in academic research.