SF-TMN: SlowFast Temporal Modeling Network for Surgical Phase
  Recognition
Bokai Zhang Mohammad Hasan Sarhan Bharti Goel Svetlana Petculescu Amer Ghanem
Automatic surgical phase recognition is one of the key technologies to
support Video-Based Assessment (VBA) systems for surgical education. Utilizing
temporal information is crucial for surgical phase recognition, hence various
recent approaches extract frame-level features to conduct full video temporal
modeling. For better temporal modeling, we propose SlowFast Temporal Modeling
Network (SF-TMN) for surgical phase recognition that can not only achieve
frame-level full video temporal modeling but also achieve segment-level full
video temporal modeling. We employ a feature extraction network, pre-trained on
the target dataset, to extract features from video frames as the training data
for SF-TMN. The Slow Path in SF-TMN utilizes all frame features for frame
temporal modeling. The Fast Path in SF-TMN utilizes segment-level features
summarized from frame features for segment temporal modeling. The proposed
paradigm is flexible regarding the choice of temporal modeling networks. We
explore MS-TCN and ASFormer models as temporal modeling networks and experiment
with multiple combination strategies for Slow and Fast Paths. We evaluate
SF-TMN on Cholec80 surgical phase recognition task and demonstrate that SF-TMN
can achieve state-of-the-art results on all considered metrics. SF-TMN with
ASFormer backbone outperforms the state-of-the-art Not End-to-End(TCN) method
by 2.6% in accuracy and 7.4% in the Jaccard score. We also evaluate SF-TMN on
action segmentation datasets including 50salads, GTEA, and Breakfast, and
achieve state-of-the-art results. The improvement in the results shows that
combining temporal information from both frame level and segment level by
refining outputs with temporal refinement stages is beneficial for the temporal
modeling of surgical phases.