Med3DInsight: Enhancing 3D Medical Image Understanding with 2D Multi-Modal Large Language Models
This work addresses a critical need in medical imaging for more efficient and accurate 3D image understanding, though it is incremental as it builds on existing 2D MLLMs and 3D encoders.
The paper tackles the problem of limited semantic understanding and high data requirements in 3D medical image analysis by proposing Med3DInsight, a pre-training framework that integrates 3D image encoders with 2D multi-modal large language models, achieving state-of-the-art performance on segmentation and classification tasks across three public datasets.
Understanding 3D medical image volumes is a critical task in the medical domain. However, existing 3D convolution and transformer-based methods have limited semantic understanding of an image volume and also need a large set of volumes for training. Recent advances in multi-modal large language models (MLLMs) provide a new and promising way to understand images with the help of text descriptions. However, most current MLLMs are designed for 2D natural images. To enhance the 3D medical image understanding with 2D MLLMs, we propose a novel pre-training framework called Med3DInsight, which marries existing 3D image encoders with 2D MLLMs and bridges them via a designed Plane-Slice-Aware Transformer (PSAT) module. Extensive experiments demonstrate our SOTA performance on two downstream segmentation and classification tasks, including three public datasets with CT and MRI modalities and comparison to more than ten baselines. Med3DInsight can be easily integrated into any current 3D medical image understanding network and improves its performance by a good margin.