Exploring Optical Flow Inclusion into nnU-Net Framework for Surgical
  Instrument Segmentation
Marcos Fernández-Rodríguez Bruno Silva Sandro Queirós Helena R. Torres Bruno Oliveira Pedro Morais Lukas R. Buschle Jorge Correia-Pinto Estevão Lima João L. Vilaça
Surgical instrument segmentation in laparoscopy is essential for
computer-assisted surgical systems. Despite the Deep Learning progress in
recent years, the dynamic setting of laparoscopic surgery still presents
challenges for precise segmentation. The nnU-Net framework excelled in semantic
segmentation analyzing single frames without temporal information. The
framework's ease of use, including its ability to be automatically configured,
and its low expertise requirements, have made it a popular base framework for
comparisons. Optical flow (OF) is a tool commonly used in video tasks to
estimate motion and represent it in a single frame, containing temporal
information. This work seeks to employ OF maps as an additional input to the
nnU-Net architecture to improve its performance in the surgical instrument
segmentation task, taking advantage of the fact that instruments are the main
moving objects in the surgical field. With this new input, the temporal
component would be indirectly added without modifying the architecture. Using
CholecSeg8k dataset, three different representations of movement were estimated
and used as new inputs, comparing them with a baseline model. Results showed
that the use of OF maps improves the detection of classes with high movement,
even when these are scarce in the dataset. To further improve performance,
future work may focus on implementing other OF-preserving augmentations.