SAM Meets Robotic Surgery: An Empirical Study on Generalization,
  Robustness and Adaptation
An Wang Mobarakol Islam Mengya Xu Yang Zhang Hongliang Ren
The Segment Anything Model (SAM) serves as a fundamental model for semantic
segmentation and demonstrates remarkable generalization capabilities across a
wide range of downstream scenarios. In this empirical study, we examine SAM's
robustness and zero-shot generalizability in the field of robotic surgery. We
comprehensively explore different scenarios, including prompted and unprompted
situations, bounding box and points-based prompt approaches, as well as the
ability to generalize under corruptions and perturbations at five severity
levels. Additionally, we compare the performance of SAM with state-of-the-art
supervised models. We conduct all the experiments with two well-known robotic
instrument segmentation datasets from MICCAI EndoVis 2017 and 2018 challenges.
Our extensive evaluation results reveal that although SAM shows remarkable
zero-shot generalization ability with bounding box prompts, it struggles to
segment the whole instrument with point-based prompts and unprompted settings.
Furthermore, our qualitative figures demonstrate that the model either failed
to predict certain parts of the instrument mask (e.g., jaws, wrist) or
predicted parts of the instrument as wrong classes in the scenario of
overlapping instruments within the same bounding box or with the point-based
prompt. In fact, SAM struggles to identify instruments in complex surgical
scenarios characterized by the presence of blood, reflection, blur, and shade.
Additionally, SAM is insufficiently robust to maintain high performance when
subjected to various forms of data corruption. We also attempt to fine-tune SAM
using Low-rank Adaptation (LoRA) and propose SurgicalSAM, which shows the
capability in class-wise mask prediction without prompt. Therefore, we can
argue that, without further domain-specific fine-tuning, SAM is not ready for
downstream surgical tasks.