CXReasonBench: A Benchmark for Evaluating Structured Diagnostic Reasoning in Chest X-rays
This addresses the problem of assessing clinically valid reasoning in medical AI for researchers and practitioners, though it is incremental as it builds on existing datasets and focuses on benchmarking rather than new methods.
The authors tackled the lack of benchmarks for evaluating structured diagnostic reasoning in chest X-rays by introducing CXReasonBench, a benchmark built on the MIMIC-CXR-JPG dataset, which revealed that even top large vision-language models struggle with clinically meaningful reasoning steps and generalization.
Recent progress in Large Vision-Language Models (LVLMs) has enabled promising applications in medical tasks, such as report generation and visual question answering. However, existing benchmarks focus mainly on the final diagnostic answer, offering limited insight into whether models engage in clinically meaningful reasoning. To address this, we present CheXStruct and CXReasonBench, a structured pipeline and benchmark built on the publicly available MIMIC-CXR-JPG dataset. CheXStruct automatically derives a sequence of intermediate reasoning steps directly from chest X-rays, such as segmenting anatomical regions, deriving anatomical landmarks and diagnostic measurements, computing diagnostic indices, and applying clinical thresholds. CXReasonBench leverages this pipeline to evaluate whether models can perform clinically valid reasoning steps and to what extent they can learn from structured guidance, enabling fine-grained and transparent assessment of diagnostic reasoning. The benchmark comprises 18,988 QA pairs across 12 diagnostic tasks and 1,200 cases, each paired with up to 4 visual inputs, and supports multi-path, multi-stage evaluation including visual grounding via anatomical region selection and diagnostic measurements. Even the strongest of 12 evaluated LVLMs struggle with structured reasoning and generalization, often failing to link abstract knowledge with anatomically grounded visual interpretation. The code is available at https://github.com/ttumyche/CXReasonBench