Real-time Surgical Instrument Segmentation in Video Using Point Tracking
  and Segment Anything
Zijian Wu Adam Schmidt Peter Kazanzides Septimiu E. Salcudean
The Segment Anything Model (SAM) is a powerful vision foundation model that
is revolutionizing the traditional paradigm of segmentation. Despite this, a
reliance on prompting each frame and large computational cost limit its usage
in robotically assisted surgery. Applications, such as augmented reality
guidance, require little user intervention along with efficient inference to be
usable clinically. In this study, we address these limitations by adopting
lightweight SAM variants to meet the speed requirement and employing
fine-tuning techniques to enhance their generalization in surgical scenes.
Recent advancements in Tracking Any Point (TAP) have shown promising results in
both accuracy and efficiency, particularly when points are occluded or leave
the field of view. Inspired by this progress, we present a novel framework that
combines an online point tracker with a lightweight SAM model that is
fine-tuned for surgical instrument segmentation. Sparse points within the
region of interest are tracked and used to prompt SAM throughout the video
sequence, providing temporal consistency. The quantitative results surpass the
state-of-the-art semi-supervised video object segmentation method on the
EndoVis 2015 dataset, with an over 25 FPS inference speed running on a single
GeForce RTX 4060 GPU.