Task-Aware Low-Rank Adaptation of Segment Anything Model
Xuehao Wang Feiyang Ye Yu Zhang
The Segment Anything Model (SAM), with its remarkable zero-shot capability,
has been proven to be a powerful foundation model for image segmentation tasks,
which is an important task in computer vision. However, the transfer of its
rich semantic information to multiple different downstream tasks remains
unexplored. In this paper, we propose the Task-Aware Low-Rank Adaptation
(TA-LoRA) method, which enables SAM to work as a foundation model for
multi-task learning. Specifically, TA-LoRA injects an update parameter tensor
into each layer of the encoder in SAM and leverages a low-rank tensor
decomposition method to incorporate both task-shared and task-specific
information. Furthermore, we introduce modified SAM (mSAM) for multi-task
learning where we remove the prompt encoder of SAM and use task-specific no
mask embeddings and mask decoder for each task. Extensive experiments conducted
on benchmark datasets substantiate the efficacy of TA-LoRA in enhancing the
performance of mSAM across multiple downstream tasks.