Ladder Fine-tuning approach for SAM integrating complementary network
Shurong Chai Rahul Kumar Jain Shiyu Teng Jiaqing Liu Yinhao Li Tomoko Tateyama Yen-wei Chen
Recently, foundation models have been introduced demonstrating various tasks
in the field of computer vision. These models such as Segment Anything Model
(SAM) are generalized models trained using huge datasets. Currently, ongoing
research focuses on exploring the effective utilization of these generalized
models for specific domains, such as medical imaging. However, in medical
imaging, the lack of training samples due to privacy concerns and other factors
presents a major challenge for applying these generalized models to medical
image segmentation task. To address this issue, the effective fine tuning of
these models is crucial to ensure their optimal utilization. In this study, we
propose to combine a complementary Convolutional Neural Network (CNN) along
with the standard SAM network for medical image segmentation. To reduce the
burden of fine tuning large foundation model and implement cost-efficient
trainnig scheme, we focus only on fine-tuning the additional CNN network and
SAM decoder part. This strategy significantly reduces trainnig time and
achieves competitive results on publicly available dataset. The code is
available at https://github.com/11yxk/SAM-LST.