MaskSAM: Towards Auto-prompt SAM with Mask Classification for Volumetric Medical Image Segmentation
This addresses the challenge of applying foundation models to medical imaging without manual prompts, offering incremental improvements for healthcare applications.
The paper tackles the problem of adapting the Segment Anything Model (SAM) for volumetric medical image segmentation by proposing MaskSAM, a prompt-free framework that generates auxiliary prompts and uses adapters for fine-tuning, achieving state-of-the-art performance with 90.52% Dice on AMOS2022 (2.7% improvement over nnUNet) and gains on other datasets.
Segment Anything Model (SAM), a prompt-driven foundation model for natural image segmentation, has demonstrated impressive zero-shot performance. However, SAM does not work when directly applied to medical image segmentation, since SAM lacks the ability to predict semantic labels, requires additional prompts, and presents suboptimal performance. Following the above issues, we propose MaskSAM, a novel mask classification prompt-free SAM adaptation framework for medical image segmentation. We design a prompt generator combined with the image encoder in SAM to generate a set of auxiliary classifier tokens, auxiliary binary masks, and auxiliary bounding boxes. Each pair of auxiliary mask and box prompts can solve the requirements of extra prompts. The semantic label prediction can be addressed by the sum of the auxiliary classifier tokens and the learnable global classifier tokens in the mask decoder of SAM. Meanwhile, we design a 3D depth-convolution adapter for image embeddings and a 3D depth-MLP adapter for prompt embeddings to efficiently fine-tune SAM. Our method achieves state-of-the-art performance on AMOS2022, 90.52% Dice, which improved by 2.7% compared to nnUNet. Our method surpasses nnUNet by 1.7% on ACDC and 1.0% on Synapse datasets.