Building a Silver-Standard Dataset from NICE Guidelines for Clinical LLMs
This provides a tool for systematically evaluating LLMs' clinical utility and guideline adherence, addressing a gap in healthcare AI, though it is incremental as it builds on existing guidelines and methods.
The study tackled the lack of standardized benchmarks for evaluating guideline-based clinical reasoning in healthcare LLMs by introducing a validated dataset derived from public guidelines, and benchmarked recent LLMs to showcase its validity.
Large language models (LLMs) are increasingly used in healthcare, yet standardised benchmarks for evaluating guideline-based clinical reasoning are missing. This study introduces a validated dataset derived from publicly available guidelines across multiple diagnoses. The dataset was created with the help of GPT and contains realistic patient scenarios, as well as clinical questions. We benchmark a range of recent popular LLMs to showcase the validity of our dataset. The framework supports systematic evaluation of LLMs' clinical utility and guideline adherence.