Dynamic 3D Point Cloud Sequences as 2D Videos
Yiming Zeng Junhui Hou Qijian Zhang Siyu Ren Wenping Wang
Dynamic 3D point cloud sequences serve as one of the most common and
practical representation modalities of dynamic real-world environments.
However, their unstructured nature in both spatial and temporal domains poses
significant challenges to effective and efficient processing. Existing deep
point cloud sequence modeling approaches imitate the mature 2D video learning
mechanisms by developing complex spatio-temporal point neighbor grouping and
feature aggregation schemes, often resulting in methods lacking effectiveness,
efficiency, and expressive power. In this paper, we propose a novel generic
representation called \textit{Structured Point Cloud Videos} (SPCVs).
Intuitively, by leveraging the fact that 3D geometric shapes are essentially 2D
manifolds, SPCV re-organizes a point cloud sequence as a 2D video with spatial
smoothness and temporal consistency, where the pixel values correspond to the
3D coordinates of points. The structured nature of our SPCV representation
allows for the seamless adaptation of well-established 2D image/video
techniques, enabling efficient and effective processing and analysis of 3D
point cloud sequences. To achieve such re-organization, we design a
self-supervised learning pipeline that is geometrically regularized and driven
by self-reconstructive and deformation field learning objectives. Additionally,
we construct SPCV-based frameworks for both low-level and high-level 3D point
cloud sequence processing and analysis tasks, including action recognition,
temporal interpolation, and compression. Extensive experiments demonstrate the
versatility and superiority of the proposed SPCV, which has the potential to
offer new possibilities for deep learning on unstructured 3D point cloud
sequences. Code will be released at https://github.com/ZENGYIMING-EAMON/SPCV.