Model2Scene: Learning 3D Scene Representation via Contrastive
  Language-CAD Models Pre-training
Runnan Chen Xinge Zhu Nenglun Chen Dawei Wang Wei Li Yuexin Ma Ruigang Yang Tongliang Liu Wenping Wang
Current successful methods of 3D scene perception rely on the large-scale
annotated point cloud, which is tedious and expensive to acquire. In this
paper, we propose Model2Scene, a novel paradigm that learns free 3D scene
representation from Computer-Aided Design (CAD) models and languages. The main
challenges are the domain gaps between the CAD models and the real scene's
objects, including model-to-scene (from a single model to the scene) and
synthetic-to-real (from synthetic model to real scene's object). To handle the
above challenges, Model2Scene first simulates a crowded scene by mixing
data-augmented CAD models. Next, we propose a novel feature regularization
operation, termed Deep Convex-hull Regularization (DCR), to project point
features into a unified convex hull space, reducing the domain gap. Ultimately,
we impose contrastive loss on language embedding and the point features of CAD
models to pre-train the 3D network. Extensive experiments verify the learned 3D
scene representation is beneficial for various downstream tasks, including
label-free 3D object salient detection, label-efficient 3D scene perception and
zero-shot 3D semantic segmentation. Notably, Model2Scene yields impressive
label-free 3D object salient detection with an average mAP of 46.08\% and
55.49\% on the ScanNet and S3DIS datasets, respectively. The code will be
publicly available.