Clinically Grounded Agent-based Report Evaluation: An Interpretable Metric for Radiology Report Generation
This addresses the need for reliable and interpretable clinical evaluation in radiology report generation, offering a domain-specific improvement over existing black-box metrics.
The paper tackled the problem of evaluating radiology report generation by introducing ICARE, an interpretable metric using large language model agents and dynamic multiple-choice question answering, which showed significant alignment with expert judgment in clinician studies.
Radiological imaging is central to diagnosis, treatment planning, and clinical decision-making. Vision-language foundation models have spurred interest in automated radiology report generation (RRG), but safe deployment requires reliable clinical evaluation of generated reports. Existing metrics often rely on surface-level similarity or behave as black boxes, lacking interpretability. We introduce ICARE (Interpretable and Clinically-grounded Agent-based Report Evaluation), an interpretable evaluation framework leveraging large language model agents and dynamic multiple-choice question answering (MCQA). Two agents, each with either the ground-truth or generated report, generate clinically meaningful questions and quiz each other. Agreement on answers captures preservation and consistency of findings, serving as interpretable proxies for clinical precision and recall. By linking scores to question-answer pairs, ICARE enables transparent, and interpretable assessment. Clinician studies show ICARE aligns significantly more with expert judgment than prior metrics. Perturbation analyses confirm sensitivity to clinical content and reproducibility, while model comparisons reveal interpretable error patterns.