UCSF-PDGM-VQA: Visual Question Answering dataset for brain tumor MRI interpretation
For neuro-oncology AI researchers, this benchmark highlights critical deficiencies in current VLMs for clinical MRI interpretation, but the work is incremental as it primarily provides a dataset and baseline evaluation.
The authors introduce a VQA benchmark for brain tumor MRI interpretation (UCSF-PDGM-VQA) with 2,387 QA pairs and find that current vision-language models fail to process multi-sequence 3D MRI scans, leading to modality collapse and over-reliance on language priors.
Brain tumor diagnosis is largely dependent on Magnetic Resonance Imaging (MRI) evaluation, which requires radiologists to synthesize thousands of images across multiple 3D sequences and longitudinal studies. This process requires advanced neuro-radiology training, poses substantial cognitive load, and is highly time-consuming. Despite increasing demands in radiology, this expertise is difficult to scale, straining the current health systems. Vision-Language Models (VLMs) provide an opportunity to reduce this burden through a semi-automated, interactive interpretation of complex brain MRIs. However, they are currently underutilized in neuro-oncology due to a lack of specialized benchmarks for evaluating them. We introduce a clinically relevant visual question answering (VQA) benchmark -- the UCSF-PDGM-VQA dataset -- consisting of 2,387 QA pairs from 473 glioma-related MRI studies in the public UCSF-PDGM dataset. We further establish a performance baseline for six state-of-the-art vision-language models (VLMs) and one large language model on this dataset. We find that current models are incapable of effectively processing multi-sequence, 3-dimensional MRI scans, thus resulting in a suppression of visual features and over-reliance on language priors, causing modality collapse. These findings underscore a critical deficiency in current model reliability and safety within clinical settings, necessitating the development of robust, domain-specific VLMs.