MedErr-CT: A Visual Question Answering Benchmark for Identifying and Correcting Errors in CT Reports
This work addresses the need for more reliable and clinically applicable medical AI systems to reduce diagnostic errors, though it is incremental as it builds on existing VQA benchmarks by adding clinical relevance.
The authors tackled the problem of diagnostic errors in CT reports by introducing MedErr-CT, a visual question answering benchmark for evaluating medical multimodal large language models' ability to identify and correct errors, revealing substantial variation in performance across error types.
Computed Tomography (CT) plays a crucial role in clinical diagnosis, but the growing demand for CT examinations has raised concerns about diagnostic errors. While Multimodal Large Language Models (MLLMs) demonstrate promising comprehension of medical knowledge, their tendency to produce inaccurate information highlights the need for rigorous validation. However, existing medical visual question answering (VQA) benchmarks primarily focus on simple visual recognition tasks, lacking clinical relevance and failing to assess expert-level knowledge. We introduce MedErr-CT, a novel benchmark for evaluating medical MLLMs' ability to identify and correct errors in CT reports through a VQA framework. The benchmark includes six error categories - four vision-centric errors (Omission, Insertion, Direction, Size) and two lexical error types (Unit, Typo) - and is organized into three task levels: classification, detection, and correction. Using this benchmark, we quantitatively assess the performance of state-of-the-art 3D medical MLLMs, revealing substantial variation in their capabilities across different error types. Our benchmark contributes to the development of more reliable and clinically applicable MLLMs, ultimately helping reduce diagnostic errors and improve accuracy in clinical practice. The code and datasets are available at https://github.com/babbu3682/MedErr-CT.