A Large-Language Model Framework for Relative Timeline Extraction from PubMed Case Reports
This work addresses the need for structured temporal data in clinical reports to support tasks like process tracing and forecasting, though it is incremental as it builds on existing LLM methods for a specific domain.
The paper tackled the problem of extracting relative timelines from clinical case reports, which lack structured temporal data, by developing a system using large language models (LLMs) to annotate events and timestamps, achieving high temporal concordance (e.g., 0.95) but moderate event recall (e.g., 0.80).
Timing of clinical events is central to characterization of patient trajectories, enabling analyses such as process tracing, forecasting, and causal reasoning. However, structured electronic health records capture few data elements critical to these tasks, while clinical reports lack temporal localization of events in structured form. We present a system that transforms case reports into textual time series-structured pairs of textual events and timestamps. We contrast manual and large language model (LLM) annotations (n=320 and n=390 respectively) of ten randomly-sampled PubMed open-access (PMOA) case reports (N=152,974) and assess inter-LLM agreement (n=3,103; N=93). We find that the LLM models have moderate event recall(O1-preview: 0.80) but high temporal concordance among identified events (O1-preview: 0.95). By establishing the task, annotation, and assessment systems, and by demonstrating high concordance, this work may serve as a benchmark for leveraging the PMOA corpus for temporal analytics.