X4D-SceneFormer: Enhanced Scene Understanding on 4D Point Cloud Videos
  through Cross-modal Knowledge Transfer
Linglin Jing Ying Xue Xu Yan Chaoda Zheng Dong Wang Ruimao Zhang Zhigang Wang Hui Fang Bin Zhao Zhen Li
The field of 4D point cloud understanding is rapidly developing with the goal
of analyzing dynamic 3D point cloud sequences. However, it remains a
challenging task due to the sparsity and lack of texture in point clouds.
Moreover, the irregularity of point cloud poses a difficulty in aligning
temporal information within video sequences. To address these issues, we
propose a novel cross-modal knowledge transfer framework, called
X4D-SceneFormer. This framework enhances 4D-Scene understanding by transferring
texture priors from RGB sequences using a Transformer architecture with
temporal relationship mining. Specifically, the framework is designed with a
dual-branch architecture, consisting of an 4D point cloud transformer and a
Gradient-aware Image Transformer (GIT). During training, we employ multiple
knowledge transfer techniques, including temporal consistency losses and masked
self-attention, to strengthen the knowledge transfer between modalities. This
leads to enhanced performance during inference using single-modal 4D point
cloud inputs. Extensive experiments demonstrate the superior performance of our
framework on various 4D point cloud video understanding tasks, including action
recognition, action segmentation and semantic segmentation. The results achieve
1st places, i.e., 85.3% (+7.9%) accuracy and 47.3% (+5.0%) mIoU for 4D action
segmentation and semantic segmentation, on the HOI4D
challenge\footnote{\url{http://www.hoi4d.top/}.}, outperforming previous
state-of-the-art by a large margin. We release the code at
https://github.com/jinglinglingling/X4D