Toward Automatic Filling of Case Report Forms: A Case Study on Data from an Italian Emergency Department
This work provides a new dataset and initial findings for researchers developing automated CRF-filling systems, particularly for the under-resourced Italian language clinical domain, which is an incremental step towards reducing manual data entry for medical professionals.
This paper addresses the scarcity of annotated Case Report Form (CRF) data by introducing a new dataset of clinical notes from an Italian Emergency Department, annotated with 134 CRF items. Pilot experiments using an open-source LLM in a zero-shot setting demonstrate the feasibility of CRF-filling from real Italian clinical notes and highlight biases in LLM responses.
Case Report Forms (CRFs) collect data about patients and are at the core of well-established practices to conduct research in clinical settings. With the recent progress of language technologies, there is an increasing interest in automatic CRF-filling from clinical notes, mostly based on the use of Large Language Models (LLMs). However, there is a general scarcity of annotated CRF data, both for training and testing LLMs, which limits the progress on this task. As a step in the direction of providing such data, we present a new dataset of clinical notes from an Italian Emergency Department annotated with respect to a pre-defined CRF containing 134 items to be filled. We provide an analysis of the data, define the CRF-filling task and metric for its evaluation, and report on pilot experiments where we use an open-source state-of-the-art LLM to automatically execute the task. Results of the case-study show that (i) CRF-filling from real clinical notes in Italian can be approached in a zero-shot setting; (ii) LLMs' results are affected by biases (e.g., a cautious behaviour favours "unknown" answers), which need to be corrected.