AgentsEval: Clinically Faithful Evaluation of Medical Imaging Reports via Multi-Agent Reasoning
This addresses the critical need for reliable and clinically relevant evaluation of medical report generation systems, particularly for integrating large language models into clinical practice, though it is incremental in improving existing evaluation methods.
The paper tackled the problem of evaluating the clinical correctness and reasoning fidelity of automatically generated medical imaging reports, which existing methods often fail to capture, by introducing AgentsEval, a multi-agent stream reasoning framework that emulates radiologists' diagnostic workflow, resulting in clinically aligned, semantically faithful, and interpretable evaluations robust under various perturbations.
Evaluating the clinical correctness and reasoning fidelity of automatically generated medical imaging reports remains a critical yet unresolved challenge. Existing evaluation methods often fail to capture the structured diagnostic logic that underlies radiological interpretation, resulting in unreliable judgments and limited clinical relevance. We introduce AgentsEval, a multi-agent stream reasoning framework that emulates the collaborative diagnostic workflow of radiologists. By dividing the evaluation process into interpretable steps including criteria definition, evidence extraction, alignment, and consistency scoring, AgentsEval provides explicit reasoning traces and structured clinical feedback. We also construct a multi-domain perturbation-based benchmark covering five medical report datasets with diverse imaging modalities and controlled semantic variations. Experimental results demonstrate that AgentsEval delivers clinically aligned, semantically faithful, and interpretable evaluations that remain robust under paraphrastic, semantic, and stylistic perturbations. This framework represents a step toward transparent and clinically grounded assessment of medical report generation systems, fostering trustworthy integration of large language models into clinical practice.