Petr Holub

Douwe J. Spaanderman, Karthik Prathaban, Petr Zelina et al.

Large language models (LLMs) are increasingly used to extract structured information from free-text clinical records, but prior work often focuses on single tasks, limited models, and English-language reports. We evaluated 15 open-weight LLMs on pathology and radiology reports across six use cases, colorectal liver metastases, liver tumours, neurodegenerative diseases, soft-tissue tumours, melanomas, and sarcomas, at three institutes in the Netherlands, UK, and Czech Republic. Models included general-purpose and medical-specialised LLMs of various sizes, and six prompting strategies were compared: zero-shot, one-shot, few-shot, chain-of-thought, self-consistency, and prompt graph. Performance was assessed using task-appropriate metrics, with consensus rank aggregation and linear mixed-effects models quantifying variance. Top-ranked models achieved macro-average scores close to inter-rater agreement across tasks. Small-to-medium general-purpose models performed comparably to large models, while tiny and specialised models performed worse. Prompt graph and few-shot prompting improved performance by ~13%. Task-specific factors, including variable complexity and annotation variability, influenced results more than model size or prompting strategy. These findings show that open-weight LLMs can extract structured data from clinical reports across diseases, languages, and institutions, offering a scalable approach for clinical data curation.

Elliot Fairweather, Rudolf Wittner, Martin Chapman et al.

Provenance templates are now a recognised methodology for the construction of data provenance records. Each template defines the provenance of a domain-specific action in abstract form, which may then be instantiated as required by a single call to the provenance template service. As data reliability and trustworthiness becomes a critical issue in an increasing number of domains, there is a corresponding need to ensure that the provenance of that data is non-repudiable. In this paper we contribute two new, complementary modules to our template model and implementation to produce non-repudiable data provenance. The first, a module that traces the operation of the provenance template service itself, and records a provenance trace of the construction of an object-level document, at the level of individual service calls. The second, a non-repudiation module that generates evidence for the data recorded about each call, annotates the service trace accordingly, and submits a representation of that evidence to a provider-agnostic notary service. We evaluate the applicability of our approach in the context of a clinical decision support system. We first define a policy to ensure the non-repudiation of evidence with respect to a security threat analysis in order to demonstrate the suitability of our solution. We then select three use cases from within a particular system, Consult, with contrasting data provenance recording requirements and analyse the subsequent performance of our prototype implementation against three different notary providers.