Evaluating Medical LLMs by Levels of Autonomy: A Survey Moving from Benchmarks to Applications
It tackles the problem of unreliable clinical deployment for medical AI practitioners by shifting focus from scores to autonomy levels, though it is incremental as a survey-based reframing.
The survey reframes evaluation of medical LLMs using a levels-of-autonomy framework (L0-L3) to address the gap between benchmark scores and safe clinical application, proposing a blueprint for risk-aware evidence and oversight.
Medical Large language models achieve strong scores on standard benchmarks; however, the transfer of those results to safe and reliable performance in clinical workflows remains a challenge. This survey reframes evaluation through a levels-of-autonomy lens (L0-L3), spanning informational tools, information transformation and aggregation, decision support, and supervised agents. We align existing benchmarks and metrics with the actions permitted at each level and their associated risks, making the evaluation targets explicit. This motivates a level-conditioned blueprint for selecting metrics, assembling evidence, and reporting claims, alongside directions that link evaluation to oversight. By centering autonomy, the survey moves the field beyond score-based claims toward credible, risk-aware evidence for real clinical use.