RefSAM3D: Adapting SAM with Cross-modal Reference for 3D Medical Image Segmentation
This work addresses the problem of accurate 3D segmentation for medical imaging tasks such as organ segmentation and tumor quantification, representing an incremental adaptation of SAM to a new domain.
The paper tackled the challenge of adapting the Segment Anything Model (SAM) from 2D natural images to 3D medical imaging like CT and MRI, achieving superior performance over state-of-the-art methods in segmenting complex anatomical structures.
The Segment Anything Model (SAM), originally built on a 2D Vision Transformer (ViT), excels at capturing global patterns in 2D natural images but struggles with 3D medical imaging modalities like CT and MRI. These modalities require capturing spatial information in volumetric space for tasks such as organ segmentation and tumor quantification. To address this challenge, we introduce RefSAM3D, which adapts SAM for 3D medical imaging by incorporating a 3D image adapter and cross-modal reference prompt generation. Our approach modifies the visual encoder to handle 3D inputs and enhances the mask decoder for direct 3D mask generation. We also integrate textual prompts to improve segmentation accuracy and consistency in complex anatomical scenarios. By employing a hierarchical attention mechanism, our model effectively captures and integrates information across different scales. Extensive evaluations on multiple medical imaging datasets demonstrate the superior performance of RefSAM3D over state-of-the-art methods. Our contributions advance the application of SAM in accurately segmenting complex anatomical structures in medical imaging.