Convolution Meets LoRA: Parameter Efficient Finetuning for Segment Anything Model
This work addresses the challenge of domain adaptation for image segmentation in specialized fields, representing an incremental improvement over existing methods.
The paper tackles the problem of adapting the Segment Anything Model (SAM) to specialized domains like medical imagery and remote sensing, where its zero-shot generalization diminishes, by introducing Conv-LoRA, a parameter-efficient fine-tuning approach that integrates convolutional parameters into Low-Rank Adaptation, resulting in improved performance across diverse benchmarks.
The Segment Anything Model (SAM) stands as a foundational framework for image segmentation. While it exhibits remarkable zero-shot generalization in typical scenarios, its advantage diminishes when applied to specialized domains like medical imagery and remote sensing. To address this limitation, this paper introduces Conv-LoRA, a simple yet effective parameter-efficient fine-tuning approach. By integrating ultra-lightweight convolutional parameters into Low-Rank Adaptation (LoRA), Conv-LoRA can inject image-related inductive biases into the plain ViT encoder, further reinforcing SAM's local prior assumption. Notably, Conv-LoRA not only preserves SAM's extensive segmentation knowledge but also revives its capacity of learning high-level image semantics, which is constrained by SAM's foreground-background segmentation pretraining. Comprehensive experimentation across diverse benchmarks spanning multiple domains underscores Conv-LoRA's superiority in adapting SAM to real-world semantic segmentation tasks.