Convolution Meets LoRA: Parameter Efficient Finetuning for Segment
  Anything Model
Zihan Zhong Zhiqiang Tang Tong He Haoyang Fang Chun Yuan
The Segment Anything Model (SAM) stands as a foundational framework for image
segmentation. While it exhibits remarkable zero-shot generalization in typical
scenarios, its advantage diminishes when applied to specialized domains like
medical imagery and remote sensing. To address this limitation, this paper
introduces Conv-LoRA, a simple yet effective parameter-efficient fine-tuning
approach. By integrating ultra-lightweight convolutional parameters into
Low-Rank Adaptation (LoRA), Conv-LoRA can inject image-related inductive biases
into the plain ViT encoder, further reinforcing SAM's local prior assumption.
Notably, Conv-LoRA not only preserves SAM's extensive segmentation knowledge
but also revives its capacity of learning high-level image semantics, which is
constrained by SAM's foreground-background segmentation pretraining.
Comprehensive experimentation across diverse benchmarks spanning multiple
domains underscores Conv-LoRA's superiority in adapting SAM to real-world
semantic segmentation tasks.