Ladder Fine-tuning approach for SAM integrating complementary network
This work addresses the problem of efficient fine-tuning for domain-specific applications like medical imaging, where data scarcity is an issue, but it is incremental as it builds on existing foundation models.
The paper tackles the challenge of applying the Segment Anything Model (SAM) to medical image segmentation with limited training data by proposing a ladder fine-tuning approach that integrates a complementary CNN and fine-tunes only the CNN and SAM decoder, reducing training time and achieving competitive results on a public dataset.
Recently, foundation models have been introduced demonstrating various tasks in the field of computer vision. These models such as Segment Anything Model (SAM) are generalized models trained using huge datasets. Currently, ongoing research focuses on exploring the effective utilization of these generalized models for specific domains, such as medical imaging. However, in medical imaging, the lack of training samples due to privacy concerns and other factors presents a major challenge for applying these generalized models to medical image segmentation task. To address this issue, the effective fine tuning of these models is crucial to ensure their optimal utilization. In this study, we propose to combine a complementary Convolutional Neural Network (CNN) along with the standard SAM network for medical image segmentation. To reduce the burden of fine tuning large foundation model and implement cost-efficient trainnig scheme, we focus only on fine-tuning the additional CNN network and SAM decoder part. This strategy significantly reduces trainnig time and achieves competitive results on publicly available dataset. The code is available at https://github.com/11yxk/SAM-LST.