CORAL: Expert-Curated medical Oncology Reports to Advance Language Model Inference
This addresses the lack of annotated datasets for evaluating LLMs in oncology, which is important for researchers and clinicians needing to extract information from cancer progress notes, though it is incremental as it builds on existing LLM methods.
The authors tackled the problem of extracting detailed oncological history from clinical notes by developing a new annotation schema and evaluating three LLMs on a corpus of 40 breast and pancreatic cancer progress notes. GPT-4 performed best with an average BLEU score of 0.73, ROUGE score of 0.72, exact-match F1-score of 0.51, and 68% accuracy on complex tasks, but further improvements are needed for reliable use.
Both medical care and observational studies in oncology require a thorough understanding of a patient's disease progression and treatment history, often elaborately documented in clinical notes. Despite their vital role, no current oncology information representation and annotation schema fully encapsulates the diversity of information recorded within these notes. Although large language models (LLMs) have recently exhibited impressive performance on various medical natural language processing tasks, due to the current lack of comprehensively annotated oncology datasets, an extensive evaluation of LLMs in extracting and reasoning with the complex rhetoric in oncology notes remains understudied. We developed a detailed schema for annotating textual oncology information, encompassing patient characteristics, tumor characteristics, tests, treatments, and temporality. Using a corpus of 40 de-identified breast and pancreatic cancer progress notes at University of California, San Francisco, we applied this schema to assess the zero-shot abilities of three recent LLMs (GPT-4, GPT-3.5-turbo, and FLAN-UL2) to extract detailed oncological history from two narrative sections of clinical progress notes. Our team annotated 9028 entities, 9986 modifiers, and 5312 relationships. The GPT-4 model exhibited overall best performance, with an average BLEU score of 0.73, an average ROUGE score of 0.72, an exact-match F1-score of 0.51, and an average accuracy of 68% on complex tasks (expert manual evaluation on subset). Notably, it was proficient in tumor characteristic and medication extraction, and demonstrated superior performance in relational inference like adverse event detection. However, further improvements are needed before using it to reliably extract important facts from cancer progress notes needed for clinical research, complex population management, and documenting quality patient care.