Video-Instrument Synergistic Network for Referring Video Instrument
  Segmentation in Robotic Surgery
Hongqiu Wang Lei Zhu Guang Yang Yike Guo Shichen Zhang Bo Xu Yueming Jin
Robot-assisted surgery has made significant progress, with instrument
segmentation being a critical factor in surgical intervention quality. It
serves as the building block to facilitate surgical robot navigation and
surgical education for the next generation of operating intelligence. Although
existing methods have achieved accurate instrument segmentation results, they
simultaneously generate segmentation masks for all instruments, without the
capability to specify a target object and allow an interactive experience. This
work explores a new task of Referring Surgical Video Instrument Segmentation
(RSVIS), which aims to automatically identify and segment the corresponding
surgical instruments based on the given language expression. To achieve this,
we devise a novel Video-Instrument Synergistic Network (VIS-Net) to learn both
video-level and instrument-level knowledge to boost performance, while previous
work only used video-level information. Meanwhile, we design a Graph-based
Relation-aware Module (GRM) to model the correlation between multi-modal
information (i.e., textual description and video frame) to facilitate the
extraction of instrument-level information. We are also the first to produce
two RSVIS datasets to promote related research. Our method is verified on these
datasets, and experimental results exhibit that the VIS-Net can significantly
outperform existing state-of-the-art referring segmentation methods. Our code
and our datasets will be released upon the publication of this work.