MedCUA-Bench: A Screenshot-Only Benchmark for Clinical Computer-Use Agents
For researchers developing AI agents for clinical workflows, this benchmark provides a reproducible testbed to evaluate and improve agent performance in medical GUI tasks.
MedCUA-Bench introduces a benchmark for clinical computer-use agents, covering 18 scenarios across 10 medical domains. The best closed-source model achieves 54.2% success, while open-source agents average 2.5%, revealing a significant gap in reliability for clinical software use.
Computer-use agents could automate repetitive screen-based clinical work, but their reliability in medical graphical user interfaces remains largely unvalidated. Existing benchmarks focus on general web or desktop tasks and underrepresent medical software, which requires domain knowledge, exhibits markedly different UI design from mainstream applications, lacks public testing environments, and demands safety validation beyond task completion. We introduce MedCUA-Bench, an interactive benchmark for clinical computer-use agents. It covers 18 clinical scenarios across 10 medical domains, reconstructed from real product manuals and open-source medical systems to capture authentic clinical interfaces while avoiding licensing and privacy constraints. Each task ships with paired intent- and step-level goals to disentangle clinical reasoning from UI execution, and is evaluated by a deterministic checker over task completion and five clinical safety dimensions. Across 23 agents, the best closed-source model reaches 54.2% strict success, while all models remain below 9% on the real OpenEMR. Open-source agents average only 2.5%, with the best reaching 16.2%. MedCUA-Bench exposes the gap between current agents and reliable clinical software use, providing a reproducible testbed for future research.