Transferring CLIP's Knowledge into Zero-Shot Point Cloud Semantic
  Segmentation
Yuanbin Wang Shaofei Huang Yulu Gao Zhen Wang Rui Wang Kehua Sheng Bo Zhang Si Liu
Traditional 3D segmentation methods can only recognize a fixed range of
classes that appear in the training set, which limits their application in
real-world scenarios due to the lack of generalization ability. Large-scale
visual-language pre-trained models, such as CLIP, have shown their
generalization ability in the zero-shot 2D vision tasks, but are still unable
to be applied to 3D semantic segmentation directly. In this work, we focus on
zero-shot point cloud semantic segmentation and propose a simple yet effective
baseline to transfer the visual-linguistic knowledge implied in CLIP to point
cloud encoder at both feature and output levels. Both feature-level and
output-level alignments are conducted between 2D and 3D encoders for effective
knowledge transfer. Concretely, a Multi-granularity Cross-modal Feature
Alignment (MCFA) module is proposed to align 2D and 3D features from global
semantic and local position perspectives for feature-level alignment. For the
output level, per-pixel pseudo labels of unseen classes are extracted using the
pre-trained CLIP model as supervision for the 3D segmentation model to mimic
the behavior of the CLIP image encoder. Extensive experiments are conducted on
two popular benchmarks of point cloud segmentation. Our method outperforms
significantly previous state-of-the-art methods under zero-shot setting (+29.2%
mIoU on SemanticKITTI and 31.8% mIoU on nuScenes), and further achieves
promising results in the annotation-free point cloud semantic segmentation
setting, showing its great potential for label-efficient learning.