Masked Spatio-Temporal Structure Prediction for Self-supervised Learning
  on Point Cloud Videos
Zhiqiang Shen Xiaoxiao Sheng Hehe Fan Longguang Wang Yulan Guo Qiong Liu Hao Wen Xi Zhou
Recently, the community has made tremendous progress in developing effective
methods for point cloud video understanding that learn from massive amounts of
labeled data. However, annotating point cloud videos is usually notoriously
expensive. Moreover, training via one or only a few traditional tasks (e.g.,
classification) may be insufficient to learn subtle details of the
spatio-temporal structure existing in point cloud videos. In this paper, we
propose a Masked Spatio-Temporal Structure Prediction (MaST-Pre) method to
capture the structure of point cloud videos without human annotations. MaST-Pre
is based on spatio-temporal point-tube masking and consists of two
self-supervised learning tasks. First, by reconstructing masked point tubes,
our method is able to capture the appearance information of point cloud videos.
Second, to learn motion, we propose a temporal cardinality difference
prediction task that estimates the change in the number of points within a
point tube. In this way, MaST-Pre is forced to model the spatial and temporal
structure in point cloud videos. Extensive experiments on MSRAction-3D,
NTU-RGBD, NvGesture, and SHREC'17 demonstrate the effectiveness of the proposed
method.