SqueezeSAM: User friendly mobile interactive segmentation
Balakrishnan Varadarajan Bilge Soran Forrest Iandola Xiaoyu Xiang Yunyang Xiong Lemeng Wu Chenchen Zhu Raghuraman Krishnamoorthi Vikas Chandra
Segment Anything Model (SAM) is a foundation model for interactive
segmentation, and it has catalyzed major advances in generative AI,
computational photography, and medical imaging. This model takes in an
arbitrary user input and provides segmentation masks of the corresponding
objects. It is our goal to develop a version of SAM that is appropriate for use
in a photography app. The original SAM model has a few challenges in this
setting. First, original SAM a 600 million parameter based on ViT-H, and its
high computational cost and large model size that are not suitable for todays
mobile hardware. We address this by proposing the SqueezeSAM model
architecture, which is 50x faster and 100x smaller than SAM. Next, when a user
takes a photo on their phone, it might not occur to them to click on the image
and get a mask. Our solution is to use salient object detection to generate the
first few clicks. This produces an initial segmentation mask that the user can
interactively edit. Finally, when a user clicks on an object, they typically
expect all related pieces of the object to be segmented. For instance, if a
user clicks on a person t-shirt in a photo, they expect the whole person to be
segmented, but SAM typically segments just the t-shirt. We address this with a
new data augmentation scheme, and the end result is that if the user clicks on
a person holding a basketball, the person and the basketball are all segmented
together.