GAPS: A Clinically Grounded, Automated Benchmark for Evaluating AI Clinicians
This provides a reproducible and scalable method for evaluating AI clinician systems to improve safety and reliability in clinical practice, though it is incremental as it builds on prior benchmark limitations.
The authors tackled the problem of inadequate benchmarks for AI clinician systems by introducing the GAPS framework, a multidimensional paradigm for evaluating grounding, adequacy, perturbation, and safety, and developed an automated pipeline to construct a benchmark, revealing that state-of-the-art models degrade sharply with increased reasoning depth, struggle with answer completeness, and are vulnerable to adversarial perturbations and safety issues.
Current benchmarks for AI clinician systems, often based on multiple-choice exams or manual rubrics, fail to capture the depth, robustness, and safety required for real-world clinical practice. To address this, we introduce the GAPS framework, a multidimensional paradigm for evaluating \textbf{G}rounding (cognitive depth), \textbf{A}dequacy (answer completeness), \textbf{P}erturbation (robustness), and \textbf{S}afety. Critically, we developed a fully automated, guideline-anchored pipeline to construct a GAPS-aligned benchmark end-to-end, overcoming the scalability and subjectivity limitations of prior work. Our pipeline assembles an evidence neighborhood, creates dual graph and tree representations, and automatically generates questions across G-levels. Rubrics are synthesized by a DeepResearch agent that mimics GRADE-consistent, PICO-driven evidence review in a ReAct loop. Scoring is performed by an ensemble of large language model (LLM) judges. Validation confirmed our automated questions are high-quality and align with clinician judgment. Evaluating state-of-the-art models on the benchmark revealed key failure modes: performance degrades sharply with increased reasoning depth (G-axis), models struggle with answer completeness (A-axis), and they are highly vulnerable to adversarial perturbations (P-axis) as well as certain safety issues (S-axis). This automated, clinically-grounded approach provides a reproducible and scalable method for rigorously evaluating AI clinician systems and guiding their development toward safer, more reliable clinical practice.