Beyond Medical Diagnostics: How Medical Multimodal Large Language Models Think in Space
This addresses a critical gap in medical AI for radiologists and researchers, though it is incremental as it focuses on benchmarking rather than solving the problem directly.
The study tackled the lack of visual spatial intelligence in Multimodal Large Language Models (MLLMs) for 3D medical imaging by introducing SpatialMed, a benchmark with nearly 10K question-answer pairs, and found that current models lack robust spatial reasoning capabilities.
Visual spatial intelligence is critical for medical image interpretation, yet remains largely unexplored in Multimodal Large Language Models (MLLMs) for 3D imaging. This gap persists due to a systemic lack of datasets featuring structured 3D spatial annotations beyond basic labels. In this study, we introduce an agentic pipeline that autonomously synthesizes spatial visual question-answering (VQA) data by orchestrating computational tools such as volume and distance calculators with multi-agent collaboration and expert radiologist validation. We present SpatialMed, the first comprehensive benchmark for evaluating 3D spatial intelligence in medical MLLMs, comprising nearly 10K question-answer pairs across multiple organs and tumor types. Our evaluations on 14 state-of-the-art MLLMs and extensive analyses reveal that current models lack robust spatial reasoning capabilities for medical imaging.