ART: Action-based Reasoning Task Benchmarking for Medical AI Agents
This work addresses the need for reliable clinical decision support by exposing failure modes in AI agents, which is incremental as it builds on existing benchmarking efforts to improve assessment in healthcare.
The paper tackled the problem of assessing medical AI agents' reasoning on action-based tasks in electronic health records by introducing the ART benchmark, which revealed substantial gaps in aggregation (28-64%) and threshold reasoning (32-38%) for models like GPT-4o-mini and Claude 3.5 Sonnet.
Reliable clinical decision support requires medical AI agents capable of safe, multi-step reasoning over structured electronic health records (EHRs). While large language models (LLMs) show promise in healthcare, existing benchmarks inadequately assess performance on action-based tasks involving threshold evaluation, temporal aggregation, and conditional logic. We introduce ART, an Action-based Reasoning clinical Task benchmark for medical AI agents, which mines real-world EHR data to create challenging tasks targeting known reasoning weaknesses. Through analysis of existing benchmarks, we identify three dominant error categories: retrieval failures, aggregation errors, and conditional logic misjudgments. Our four-stage pipeline -- scenario identification, task generation, quality audit, and evaluation -- produces diverse, clinically validated tasks grounded in real patient data. Evaluating GPT-4o-mini and Claude 3.5 Sonnet on 600 tasks shows near-perfect retrieval after prompt refinement, but substantial gaps in aggregation (28--64%) and threshold reasoning (32--38%). By exposing failure modes in action-oriented EHR reasoning, ART advances toward more reliable clinical agents, an essential step for AI systems that reduce cognitive load and administrative burden, supporting workforce capacity in high-demand care settings