Jointly Extracting Interventions, Outcomes, and Findings from RCT Reports with LLMs
This work addresses the onerous manual process for clinicians extracting findings from RCT reports, offering a semi-automated solution with significant performance improvements, though it is incremental as it builds on existing LLM methods for a known bottleneck in evidence extraction.
The paper tackled the problem of manually extracting structured evidence from randomized controlled trial (RCT) reports by proposing a text-to-text model based on instruction-tuned large language models (LLMs) to jointly extract interventions, outcomes, and comparators (ICO elements) and infer results, achieving approximately 20-point absolute F1 score gains over the previous state-of-the-art.
Results from Randomized Controlled Trials (RCTs) establish the comparative effectiveness of interventions, and are in turn critical inputs for evidence-based care. However, results from RCTs are presented in (often unstructured) natural language articles describing the design, execution, and outcomes of trials; clinicians must manually extract findings pertaining to interventions and outcomes of interest from such articles. This onerous manual process has motivated work on (semi-)automating extraction of structured evidence from trial reports. In this work we propose and evaluate a text-to-text model built on instruction-tuned Large Language Models (LLMs) to jointly extract Interventions, Outcomes, and Comparators (ICO elements) from clinical abstracts, and infer the associated results reported. Manual (expert) and automated evaluations indicate that framing evidence extraction as a conditional generation task and fine-tuning LLMs for this purpose realizes considerable ($\sim$20 point absolute F1 score) gains over the previous SOTA. We perform ablations and error analyses to assess aspects that contribute to model performance, and to highlight potential directions for further improvements. We apply our model to a collection of published RCTs through mid-2022, and release a searchable database of structured findings: http://ico-relations.ebm-nlp.com