Adverse Event Extraction from Discharge Summaries: A New Dataset, Annotation Scheme, and Initial Findings
This work addresses a gap in clinical NLP for underrepresented elderly populations, though it is incremental as it focuses on dataset creation and benchmarking.
The authors tackled the problem of extracting adverse events from discharge summaries for elderly patients, creating a new annotated dataset with complex entity types and evaluating models that achieved high performance on coarse-grained tasks (F1 = 0.943) but struggled with fine-grained extraction (F1 = 0.675).
In this work, we present a manually annotated corpus for Adverse Event (AE) extraction from discharge summaries of elderly patients, a population often underrepresented in clinical NLP resources. The dataset includes 14 clinically significant AEs-such as falls, delirium, and intracranial haemorrhage, along with contextual attributes like negation, diagnosis type, and in-hospital occurrence. Uniquely, the annotation schema supports both discontinuous and overlapping entities, addressing challenges rarely tackled in prior work. We evaluate multiple models using FlairNLP across three annotation granularities: fine-grained, coarse-grained, and coarse-grained with negation. While transformer-based models (e.g., BERT-cased) achieve strong performance on document-level coarse-grained extraction (F1 = 0.943), performance drops notably for fine-grained entity-level tasks (e.g., F1 = 0.675), particularly for rare events and complex attributes. These results demonstrate that despite high-level scores, significant challenges remain in detecting underrepresented AEs and capturing nuanced clinical language. Developed within a Trusted Research Environment (TRE), the dataset is available upon request via DataLoch and serves as a robust benchmark for evaluating AE extraction methods and supporting future cross-dataset generalisation.