ELM: Ensemble of Language Models for Predicting Tumor Group from Pathology Reports
This addresses a domain-specific problem for population-based cancer registries by automating tumor group classification, though it is incremental as it builds on existing language model methods.
The paper tackled the bottleneck of manually extracting tumor group data from pathology reports in cancer registries, which consumes 900 person-hours per 100,000 reports, and introduced ELM, an ensemble of language models that achieved an average precision and recall of 0.94, saving hundreds of person-hours annually.
Population-based cancer registries (PBCRs) face a significant bottleneck in manually extracting data from unstructured pathology reports, a process crucial for tasks like tumor group assignment, which can consume 900 person-hours for approximately 100,000 reports. To address this, we introduce ELM (Ensemble of Language Models), a novel ensemble-based approach leveraging both small language models (SLMs) and large language models (LLMs). ELM utilizes six fine-tuned SLMs, where three SLMs use the top part of the pathology report and three SLMs use the bottom part. This is done to maximize report coverage. ELM requires five-out-of-six agreement for a tumor group classification. Disagreements are arbitrated by an LLM with a carefully curated prompt. Our evaluation across nineteen tumor groups demonstrates ELM achieves an average precision and recall of 0.94, outperforming single-model and ensemble-without-LLM approaches. Deployed at the British Columbia Cancer Registry, ELM demonstrates how LLMs can be successfully applied in a PBCR setting to achieve state-of-the-art results and significantly enhance operational efficiencies, saving hundreds of person-hours annually.