Towards Large-scale 3D Representation Learning with Multi-dataset Point
  Prompt Training
Xiaoyang Wu Zhuotao Tian Xin Wen Bohao Peng Xihui Liu Kaicheng Yu Hengshuang Zhao
The rapid advancement of deep learning models often attributes to their
ability to leverage massive training data. In contrast, such privilege has not
yet fully benefited 3D deep learning, mainly due to the limited availability of
large-scale 3D datasets. Merging multiple available data sources and letting
them collaboratively train a single model is a potential solution. However, due
to the large domain gap between 3D point cloud datasets, such mixed supervision
could adversely affect the model's performance and lead to degenerated
performance (i.e., negative transfer) compared to single-dataset training. In
view of this challenge, we introduce Point Prompt Training (PPT), a novel
framework for multi-dataset synergistic learning in the context of 3D
representation learning that supports multiple pre-training paradigms. Based on
this framework, we propose Prompt-driven Normalization, which adapts the model
to different datasets with domain-specific prompts and Language-guided
Categorical Alignment that decently unifies the multiple-dataset label spaces
by leveraging the relationship between label text. Extensive experiments verify
that PPT can overcome the negative transfer associated with synergistic
learning and produce generalizable representations. Notably, it achieves
state-of-the-art performance on each dataset using a single weight-shared model
with supervised multi-dataset training. Moreover, when served as a pre-training
framework, it outperforms other pre-training approaches regarding
representation quality and attains remarkable state-of-the-art performance
across over ten diverse downstream tasks spanning both indoor and outdoor 3D
scenarios.