Cheap Lunch for Medical Image Segmentation by Fine-tuning SAM on Few
  Exemplars
Weijia Feng Lingting Zhu Lequan Yu
The Segment Anything Model (SAM) has demonstrated remarkable capabilities of
scaled-up segmentation models, enabling zero-shot generalization across a
variety of domains. By leveraging large-scale foundational models as
pre-trained models, it is a natural progression to fine-tune SAM for specific
domains to further enhance performances. However, the adoption of foundational
models in the medical domain presents a challenge due to the difficulty and
expense of labeling sufficient data for adaptation within hospital systems. In
this paper, we introduce an efficient and practical approach for fine-tuning
SAM using a limited number of exemplars, making it suitable for such scenarios.
Our approach combines two established techniques from the literature: an
exemplar-guided synthesis module and the widely recognized Low-Rank Adaptation
(LoRA) fine-tuning strategy, serving as data-level and model-level attempts
respectively. Interestingly, our empirical findings suggest that SAM can be
effectively aligned within the medical domain even with few labeled data. We
validate our approach through experiments on brain tumor segmentation (BraTS)
and multi-organ CT segmentation (Synapse). The comprehensive results underscore
the feasibility and effectiveness of such an approach, paving the way for the
practical application of SAM in the medical domain.