Customizing Segmentation Foundation Model via Prompt Learning for
  Instance Segmentation
Hyung-Il Kim Kimin Yun Jun-Seok Yun Yuseok Bae
Recently, foundation models trained on massive datasets to adapt to a wide
range of domains have attracted considerable attention and are actively being
explored within the computer vision community. Among these, the Segment
Anything Model (SAM) stands out for its remarkable progress in generalizability
and flexibility for image segmentation tasks, achieved through prompt-based
object mask generation. However, despite its strength, SAM faces two key
limitations when applied to customized instance segmentation that segments
specific objects or those in unique environments not typically present in the
training data: 1) the ambiguity inherent in input prompts and 2) the necessity
for extensive additional training to achieve optimal segmentation. To address
these challenges, we propose a novel method, customized instance segmentation
via prompt learning tailored to SAM. Our method involves a prompt learning
module (PLM), which adjusts input prompts into the embedding space to better
align with user intentions, thereby enabling more efficient training.
Furthermore, we introduce a point matching module (PMM) to enhance the feature
representation for finer segmentation by ensuring detailed alignment with
ground truth boundaries. Experimental results on various customized instance
segmentation scenarios demonstrate the effectiveness of the proposed method.