From Generalization to Precision: Exploring SAM for Tool Segmentation in
  Surgical Environments
Kanyifeechukwu J. Oguine Roger D. Soberanis-Mukul Nathan Drenkow Mathias Unberath
Purpose: Accurate tool segmentation is essential in computer-aided
procedures. However, this task conveys challenges due to artifacts' presence
and the limited training data in medical scenarios. Methods that generalize to
unseen data represent an interesting venue, where zero-shot segmentation
presents an option to account for data limitation. Initial exploratory works
with the Segment Anything Model (SAM) show that bounding-box-based prompting
presents notable zero-short generalization. However, point-based prompting
leads to a degraded performance that further deteriorates under image
corruption. We argue that SAM drastically over-segment images with high
corruption levels, resulting in degraded performance when only a single
segmentation mask is considered, while the combination of the masks overlapping
the object of interest generates an accurate prediction. Method: We use SAM to
generate the over-segmented prediction of endoscopic frames. Then, we employ
the ground-truth tool mask to analyze the results of SAM when the best single
mask is selected as prediction and when all the individual masks overlapping
the object of interest are combined to obtain the final predicted mask. We
analyze the Endovis18 and Endovis17 instrument segmentation datasets using
synthetic corruptions of various strengths and an In-House dataset featuring
counterfactually created real-world corruptions. Results: Combining the
over-segmented masks contributes to improvements in the IoU. Furthermore,
selecting the best single segmentation presents a competitive IoU score for
clean images. Conclusions: Combined SAM predictions present improved results
and robustness up to a certain corruption level. However, appropriate prompting
strategies are fundamental for implementing these models in the medical domain.