SimAda: A Simple Unified Framework for Adapting Segment Anything Model
  in Underperformed Scenes
Yiran Song Qianyu Zhou Xuequan Lu Zhiwen Shao Lizhuang Ma
Segment anything model (SAM) has demonstrated excellent generalization
capabilities in common vision scenarios, yet lacking an understanding of
specialized data. Although numerous works have focused on optimizing SAM for
downstream tasks, these task-specific approaches usually limit the
generalizability to other downstream tasks. In this paper, we aim to
investigate the impact of the general vision modules on finetuning SAM and
enable them to generalize across all downstream tasks. We propose a simple
unified framework called SimAda for adapting SAM in underperformed scenes.
Specifically, our framework abstracts the general modules of different methods
into basic design elements, and we design four variants based on a shared
theoretical framework. SimAda is simple yet effective, which removes all
dataset-specific designs and focuses solely on general optimization, ensuring
that SimAda can be applied to all SAM-based and even Transformer-based models.
We conduct extensive experiments on nine datasets of six downstream tasks. The
results demonstrate that SimAda significantly improves the performance of SAM
on multiple downstream tasks and achieves state-of-the-art performance on most
of them, without requiring task-specific designs. Code is available at:
https://github.com/zongzi13545329/SimAda