ClinDEF: A Dynamic Evaluation Framework for Large Language Models in Clinical Reasoning
This addresses the need for better evaluation of LLMs in clinical settings, though it is incremental as it builds on existing dynamic medical frameworks.
The authors tackled the problem of evaluating large language models (LLMs) in clinical reasoning by proposing ClinDEF, a dynamic framework that simulates diagnostic dialogues, and the result was that it effectively exposed critical reasoning gaps in state-of-the-art LLMs, offering a more nuanced evaluation paradigm.
Clinical diagnosis begins with doctor-patient interaction, during which physicians iteratively gather information, determine examination and refine differential diagnosis through patients' response. This dynamic clinical-reasoning process is poorly represented by existing LLM benchmarks that focus on static question-answering. To mitigate these gaps, recent methods explore dynamic medical frameworks involving interactive clinical dialogues. Although effective, they often rely on limited, contamination-prone datasets and lack granular, multi-level evaluation. In this work, we propose ClinDEF, a dynamic framework for assessing clinical reasoning in LLMs through simulated diagnostic dialogues. Grounded in a disease knowledge graph, our method dynamically generates patient cases and facilitates multi-turn interactions between an LLM-based doctor and an automated patient agent. Our evaluation protocol goes beyond diagnostic accuracy by incorporating fine-grained efficiency analysis and rubric-based assessment of diagnostic quality. Experiments show that ClinDEF effectively exposes critical clinical reasoning gaps in state-of-the-art LLMs, offering a more nuanced and clinically meaningful evaluation paradigm.