Human-in-the-Loop Large Language Model Framework for Identification of Cutaneous Immune-Related Adverse Events
For clinicians and researchers extracting adverse event data from clinical notes, this framework offers a scalable and accurate alternative to manual review, though it is an incremental application of existing LLM methods to a specific domain.
This study developed a retrieval-augmented, multi-agent LLM framework with human-in-the-loop for detecting cutaneous immune-related adverse events from clinical notes, achieving higher accuracy (F1=0.88 vs 0.77), better inter-rater agreement (kappa=0.82 vs 0.50), and halving review time compared to manual review.
This study evaluated a retrieval-augmented, multi-agent large language model (LLM)-driven, human-in-the-loop framework for detecting cutaneous immune-related adverse events (cirAEs) from clinical notes. Compared with unassisted manual review, the LLM-assisted workflow improved accuracy (F1 = 0.88 vs 0.77), inter-rater agreement measured by Cohen's kappa (kappa = 0.82 vs 0.50), and reduced average review time by approximately half. This framework pilots how LLMs can be applied to identify immune-related toxicities across organ systems and, more broadly, enable accurate, scalable, and transparent adverse event data extraction.