SAM-PARSER: Fine-tuning SAM Efficiently by Parameter Space Reconstruction
This work addresses the problem of efficient adaptation of large segmentation models for varied scenarios, representing an incremental improvement in parameter-efficient fine-tuning.
The paper tackles the challenge of fine-tuning the Segment Anything Model (SAM) for diverse downstream segmentation tasks by proposing SAM-PARSER, which reconstructs parameter spaces using nearly zero trainable parameters, achieving superior performance while reducing trainable parameters by approximately 290 times compared to existing methods.
Segment Anything Model (SAM) has received remarkable attention as it offers a powerful and versatile solution for object segmentation in images. However, fine-tuning SAM for downstream segmentation tasks under different scenarios remains a challenge, as the varied characteristics of different scenarios naturally requires diverse model parameter spaces. Most existing fine-tuning methods attempt to bridge the gaps among different scenarios by introducing a set of new parameters to modify SAM's original parameter space. Unlike these works, in this paper, we propose fine-tuning SAM efficiently by parameter space reconstruction (SAM-PARSER), which introduce nearly zero trainable parameters during fine-tuning. In SAM-PARSER, we assume that SAM's original parameter space is relatively complete, so that its bases are able to reconstruct the parameter space of a new scenario. We obtain the bases by matrix decomposition, and fine-tuning the coefficients to reconstruct the parameter space tailored to the new scenario by an optimal linear combination of the bases. Experimental results show that SAM-PARSER exhibits superior segmentation performance across various scenarios, while reducing the number of trainable parameters by $\approx 290$ times compared with current parameter-efficient fine-tuning methods.