MedArena: Comparing LLMs for Medicine-in-the-Wild Clinician Preferences
This addresses the gap between benchmark performance and clinical utility for medical LLMs, providing a more realistic evaluation method for clinicians.
The researchers tackled the problem that current evaluation methods for medical LLMs rely on static benchmarks that don't capture real-world clinical complexity, by creating MedArena, an interactive platform where clinicians compare LLM responses to their own queries. They found that Gemini 2.0 Flash Thinking, Gemini 2.5 Pro, and GPT-4o were the top three models out of 12 tested, with only one-third of clinician questions resembling factual recall tasks and clinicians prioritizing depth, detail, and clarity over raw factual accuracy.
Large language models (LLMs) are increasingly central to clinician workflows, spanning clinical decision support, medical education, and patient communication. However, current evaluation methods for medical LLMs rely heavily on static, templated benchmarks that fail to capture the complexity and dynamics of real-world clinical practice, creating a dissonance between benchmark performance and clinical utility. To address these limitations, we present MedArena, an interactive evaluation platform that enables clinicians to directly test and compare leading LLMs using their own medical queries. Given a clinician-provided query, MedArena presents responses from two randomly selected models and asks the user to select the preferred response. Out of 1571 preferences collected across 12 LLMs up to November 1, 2025, Gemini 2.0 Flash Thinking, Gemini 2.5 Pro, and GPT-4o were the top three models by Bradley-Terry rating. Only one-third of clinician-submitted questions resembled factual recall tasks (e.g., MedQA), whereas the majority addressed topics such as treatment selection, clinical documentation, or patient communication, with ~20% involving multi-turn conversations. Additionally, clinicians cited depth and detail and clarity of presentation more often than raw factual accuracy when explaining their preferences, highlighting the importance of readability and clinical nuance. We also confirm that the model rankings remain stable even after controlling for style-related factors like response length and formatting. By grounding evaluation in real-world clinical questions and preferences, MedArena offers a scalable platform for measuring and improving the utility and efficacy of medical LLMs.