End-to-End Streaming Video Temporal Action Segmentation with Reinforce
  Learning
Wujun Wen Jinrong Zhang Shenglan Liu Yunheng Li Qifeng Li Lin Feng
Temporal Action Segmentation (TAS) from video is a kind of frame recognition
task for long video with multiple action classes. As an video understanding
task for long videos, current methods typically combine multi-modality action
recognition models with temporal models to convert feature sequences to label
sequences. This approach can only be applied to offline scenarios, which
severely limits the TAS application. Therefore, this paper proposes an
end-to-end Streaming Video Temporal Action Segmentation with Reinforce Learning
(SVTAS-RL). The end-to-end SVTAS which regard TAS as an action segment
clustering task can expand the application scenarios of TAS; and RL is used to
alleviate the problem of inconsistent optimization objective and direction.
Through extensive experiments, the SVTAS-RL model achieves a competitive
performance to the state-of-the-art model of TAS on multiple datasets, and
shows greater advantages on the ultra-long video dataset EGTEA. This indicates
that our method can replace all current TAS models end-to-end and SVTAS-RL is
more suitable for long video TAS. Code is availabel at
https://github.com/Thinksky5124/SVTAS.