Parameter Efficient Fine-Tuning of Segment Anything Model for Biomedical Imaging
This work addresses the problem of reducing computational costs for fine-tuning vision foundation models in biomedical imaging, offering an incremental improvement in efficiency.
The study tackled the challenge of efficiently fine-tuning the Segment Anything Model for biomedical image segmentation by exploring parameter-efficient fine-tuning (PEFT) methods, finding that layer placement is more critical than layer type for vision transformers and providing a recipe for resource-efficient fine-tuning.
Segmentation is an important analysis task for biomedical images, enabling the study of individual organelles, cells or organs. Deep learning has massively improved segmentation methods, but challenges remain in generalization to new conditions, requiring costly data annotation. Vision foundation models, such as Segment Anything Model (SAM), address this issue through improved generalization. However, these models still require finetuning on annotated data, although with less annotations, to achieve optimal results for new conditions. As a downside, they require more computational resources. This makes parameter-efficient finetuning (PEFT) relevant. We contribute the first comprehensive study of PEFT for SAM applied to biomedical images. We find that the placement of PEFT layers is more important for efficiency than the type of layer for vision transformers and we provide a recipe for resource-efficient finetuning. Our code is publicly available at https://github.com/computational-cell-analytics/peft-sam.