M3CoTBench: Benchmark Chain-of-Thought of MLLMs in Medical Image Understanding
This work addresses the problem of opaque AI reasoning in medical diagnosis for healthcare professionals, but it is incremental as it builds on existing chain-of-thought paradigms by applying them to a new domain-specific benchmark.
The paper tackles the lack of benchmarks for evaluating chain-of-thought reasoning in medical image understanding by introducing M3CoTBench, a new benchmark that assesses correctness, efficiency, impact, and consistency across 24 examination types and 13 tasks, revealing limitations in current multimodal large language models for generating reliable clinical reasoning.
Chain-of-Thought (CoT) reasoning has proven effective in enhancing large language models by encouraging step-by-step intermediate reasoning, and recent advances have extended this paradigm to Multimodal Large Language Models (MLLMs). In the medical domain, where diagnostic decisions depend on nuanced visual cues and sequential reasoning, CoT aligns naturally with clinical thinking processes. However, Current benchmarks for medical image understanding generally focus on the final answer while ignoring the reasoning path. An opaque process lacks reliable bases for judgment, making it difficult to assist doctors in diagnosis. To address this gap, we introduce a new M3CoTBench benchmark specifically designed to evaluate the correctness, efficiency, impact, and consistency of CoT reasoning in medical image understanding. M3CoTBench features 1) a diverse, multi-level difficulty dataset covering 24 examination types, 2) 13 varying-difficulty tasks, 3) a suite of CoT-specific evaluation metrics (correctness, efficiency, impact, and consistency) tailored to clinical reasoning, and 4) a performance analysis of multiple MLLMs. M3CoTBench systematically evaluates CoT reasoning across diverse medical imaging tasks, revealing current limitations of MLLMs in generating reliable and clinically interpretable reasoning, and aims to foster the development of transparent, trustworthy, and diagnostically accurate AI systems for healthcare. Project page at https://juntaojianggavin.github.io/projects/M3CoTBench/.