Action segmentation as a milestone towards building automatic systems to understand untrimmed videos has received considerable attention in the recent years. It is typically being modeled as a sequence labeling problem but contains intrinsic and sufficient differences than text parsing or speech processing. In this paper, we introduce a novel hybrid temporal convolutional and recurrent network (TricorNet), which has an encoder-decoder architecture: the encoder consists of a hierarchy of temporal convolutional kernels that capture the local motion changes of different actions; the decoder is a hierarchy of recurrent neural networks that are able to learn and memorize long-term action dependencies after the encoding stage. Our model is simple but extremely effective in terms of video sequence labeling. The experimental results on three public action segmentation datasets have shown that the proposed model achieves superior performance over the state of the art. Action segmentation is a challenging problem in high-level video understanding. In its simplest form, action segmentation aims to segment a temporally untrimmed video by time and label each segmented part with one of k pre-defined action labels. For example, given a video of Making Hotdog (see FIG0 ), we label the first 10 seconds as take bread, and the next 20 seconds as take sausage, and the remaining video as pour ketchup following the procedure dependencies of making a hotdog. The results of action segmentation can be further used as input to various applications, such as video-to-text BID2 and action localization BID14 .Most current approaches for action segmentation BID26 BID19 use features extracted by convolutional neural networks, e.g., two-stream CNNs BID18 or local 3D ConvNets BID23 , at every frame after a downsampling as the input, and apply a one-dimensional sequence prediction model, such as recurrent neural networks, to label actions on frames. Despite the simplicity in handling video data, action segmentation is treated similar to text parsing BID1 , which results the local motion changes in various actions being under-explored. For example , the action pour ketchup may consist of a series of sub-actions, e.g., pick up the ketchup, squeeze and pour, and put down the ketchup. Furthermore , the time duration of performing the same action pour ketchup may vary according to different people and contexts.Indeed, the recent work by BID12 starts to explore the local motion changes in action segmentation. They propose an encoder-decoder framework, similar to the deconvolution networks in image semantic segmentation BID15 , for video sequence labeling. By using a hierarchy of 1D temporal convolutional and deconvolutional kernels in the encoder and decoder networks, respectively, their model is effective in terms of capturing the local motions and achieves state-of-theart performance in various action segmentation datasets. However, one obvious drawback is that it fails to capture the long-term dependencies of different actions in a video due to its fixed-size, local receptive fields. For example, pour ketchup usually happens after both take bread and take sausage for a typically video of Making Hotdog. In addition, a dilated temporal convolutional network, similar to the WavNet for speech processing BID24 , is also tested in BID12 , but has worse performance, which further suggests the existence of differences between video and speech data, despite they are both being represented as sequential features. To overcome the above limitations , we propose a novel hybrid TempoRal COnvolutional and Recurrent Network (TricorNet), that attends to both local motion changes and long-term action dependencies for modeling video action segmentation. TricorNet uses frame-level features as the input to an encoder-decoder architecture. The encoder is a temporal convolutional network that consists of a hierarchy of one-dimensional convolutional kernels, observing that the convolutional kernels are good at encoding the local motion changes; the decoder is a hierarchy of recurrent neural networks, in our case Bi-directional Long Short-Term Memory networks (Bi-LSTMs) BID5 , that are able to learn and memorize long-term action dependencies after the encoding process. Our network is simple but extremely effective in terms of dealing with different time durations of actions and modeling the dependencies among different actions.We conduct extensive experiments on three public action segmentation datasets, where we compare our proposed models with a set of recent action segmentation networks using three different evaluation metrics. The quantitative experimental results show that our proposed TricorNet achieves superior or competitive performance to state of the art on all three datasets. A further qualitative exploration on action dependencies shows that our model is good at capturing long-term action dependencies and produce smoother labeling.For the rest of the paper, we first survey related work in the domain of action segmentation and action detection in Sec. 2. We introduce our hybrid temporal convolutional and recurrent network with some implementation variants in Sec. 3. We present both quantitative and qualitative experimental results in Sec. 4, and conclude the paper in Sec. 5. In this paper, we propose TricorNet, a novel hybrid temporal convolutional and recurrent network for video action segmentation problems. Taking frame-level features as the input to an encoder-decoder architecture, TricorNet uses temporal convolutional kernels to model local motion changes and uses bi-directional LSTM units to learn long-term action dependencies. We provide three model variants to comprehensively evaluate our model design. Despite the simplicity in methods, experimental results on three public action segmentation datasets with different metrics show that our proposed model achieves superior performance over the state of the art. A further qualitative exploration on action dependencies shows that our model is good at capturing long-term action dependencies, which help to produce segmentation in a smoother and preciser manner.Limitations. In experiments we find that all the best results of TricorNet are achieved with number of layers K = 2. It will either over-fit or stuck in local optimum when adding more layers. Considering all three datasets are relatively small with limited training data (despite they are standard in evaluating action segmentation), using more data is likely going to further improve the performance.Future Work. We consider two directions for the future work. Firstly, the proposed TricorNet is good to be evaluated on other action segmentation datasets to further explore its strengths and limitations. Secondly, TricorNet can be extended to solve other video understanding problems, taking advantage of its flexible structural design and superior capability of capturing video information.
We propose a new hybrid temporal network that achieves state-of-the-art performance on video action segmentation on three public datasets. Discusses the problem of action segmentation in long videos, up to 10 minutes long by using a temporal convolutional encoder-decoder architecture Proposes a combination of temporal convolutional and recurrent network for video action segmentation