Task-Aware Low-Rank Adaptation of Segment Anything Model
This work addresses the challenge of adapting a powerful foundation model for multi-task learning in computer vision, representing an incremental improvement.
The paper tackles the problem of transferring the Segment Anything Model (SAM) to multiple downstream segmentation tasks by proposing Task-Aware Low-Rank Adaptation (TA-LoRA) and a modified SAM (mSAM), resulting in enhanced performance across benchmark datasets.
The Segment Anything Model (SAM), with its remarkable zero-shot capability, has been proven to be a powerful foundation model for image segmentation tasks, which is an important task in computer vision. However, the transfer of its rich semantic information to multiple different downstream tasks remains unexplored. In this paper, we propose the Task-Aware Low-Rank Adaptation (TA-LoRA) method, which enables SAM to work as a foundation model for multi-task learning. Specifically, TA-LoRA injects an update parameter tensor into each layer of the encoder in SAM and leverages a low-rank tensor decomposition method to incorporate both task-shared and task-specific information. Furthermore, we introduce modified SAM (mSAM) for multi-task learning where we remove the prompt encoder of SAM and use task-specific no mask embeddings and mask decoder for each task. Extensive experiments conducted on benchmark datasets substantiate the efficacy of TA-LoRA in enhancing the performance of mSAM across multiple downstream tasks.