Brain3D: Brain Report Automation via Inflated Vision Transformers in 3D
This work addresses the need for accurate neuroradiological interpretation in medical imaging, though it is incremental as it builds on existing vision-language models with domain-specific tailoring.
The paper tackled the problem of generating automated radiology reports from 3D brain MRI scans by developing Brain3D, a vision-language framework that uses an inflated 3D architecture and staged alignment, achieving a Clinical Pathology F1 of 0.951 compared to 0.413 for a 2D baseline.
Current medical vision-language models (VLMs) process volumetric brain MRI using 2D slice-based approximations, fragmenting the spatial context required for accurate neuroradiological interpretation. We developed \textbf{Brain3D}, a staged vision-language framework for automated radiology report generation from 3D brain tumor MRI. Our approach inflates a pretrained 2D medical encoder into a native 3D architecture and progressively aligns it with a causal language model through three stages: contrastive grounding, supervised projector warmup, and LoRA-based linguistic specialization. Unlike generalist 3D medical VLMs, \textbf{Brain3D} is tailored to neuroradiology, where hemispheric laterality, tumor infiltration patterns, and anatomical localization are critical. Evaluated on 468 subjects (BraTS pathological cases plus healthy controls), our model achieves a Clinical Pathology F1 of 0.951 versus 0.413 for a strong 2D baseline while maintaining perfect specificity on healthy scans. The staged alignment proves essential: contrastive grounding establishes visual-textual correspondence, projector warmup stabilizes conditioning, and LoRA adaptation shifts output from verbose captions to structured clinical reports\footnote{Our code is publicly available for transparency and reproducibility