SAM Fewshot Finetuning for Anatomical Segmentation in Medical Images
This addresses the problem of time-consuming manual labeling for medical image segmentation, particularly for volumetric images, though it appears incremental as it builds on SAM with a novel fine-tuning approach.
The authors tackled anatomical segmentation in medical images by proposing a few-shot fine-tuning strategy for Segment Anything (SAM) that reformulates the mask decoder to use embeddings from limited labeled slices as prompts, achieving approximately 50% improvement in IoU over SAM with point prompts and performing on-par with fully supervised methods while reducing labeled data needs by at least an order of magnitude.
We propose a straightforward yet highly effective few-shot fine-tuning strategy for adapting the Segment Anything (SAM) to anatomical segmentation tasks in medical images. Our novel approach revolves around reformulating the mask decoder within SAM, leveraging few-shot embeddings derived from a limited set of labeled images (few-shot collection) as prompts for querying anatomical objects captured in image embeddings. This innovative reformulation greatly reduces the need for time-consuming online user interactions for labeling volumetric images, such as exhaustively marking points and bounding boxes to provide prompts slice by slice. With our method, users can manually segment a few 2D slices offline, and the embeddings of these annotated image regions serve as effective prompts for online segmentation tasks. Our method prioritizes the efficiency of the fine-tuning process by exclusively training the mask decoder through caching mechanisms while keeping the image encoder frozen. Importantly, this approach is not limited to volumetric medical images, but can generically be applied to any 2D/3D segmentation task. To thoroughly evaluate our method, we conducted extensive validation on four datasets, covering six anatomical segmentation tasks across two modalities. Furthermore, we conducted a comparative analysis of different prompting options within SAM and the fully-supervised nnU-Net. The results demonstrate the superior performance of our method compared to SAM employing only point prompts (approximately 50% improvement in IoU) and performs on-par with fully supervised methods whilst reducing the requirement of labeled data by at least an order of magnitude.