O-TALC: Steps Towards Combating Oversegmentation within Online Action
  Segmentation
Matthew Kent Myers Nick Wright A. Stephen McGough Nicholas Martin
Online temporal action segmentation shows a strong potential to facilitate
many HRI tasks where extended human action sequences must be tracked and
understood in real time. Traditional action segmentation approaches, however,
operate in an offline two stage approach, relying on computationally expensive
video wide features for segmentation, rendering them unsuitable for online HRI
applications. In order to facilitate online action segmentation on a stream of
incoming video data, we introduce two methods for improved training and
inference of backbone action recognition models, allowing them to be deployed
directly for online frame level classification. Firstly, we introduce surround
dense sampling whilst training to facilitate training vs. inference clip
matching and improve segment boundary predictions. Secondly, we introduce an
Online Temporally Aware Label Cleaning (O-TALC) strategy to explicitly reduce
oversegmentation during online inference. As our methods are backbone
invariant, they can be deployed with computationally efficient spatio-temporal
action recognition models capable of operating in real time with a small
segmentation latency. We show our method outperforms similar online action
segmentation work as well as matches the performance of many offline models
with access to full temporal resolution when operating on challenging
fine-grained datasets.