Generalized Label-Efficient 3D Scene Parsing via Hierarchical Feature
  Aligned Pre-Training and Region-Aware Fine-tuning
Kangcheng Liu Yong-Jin Liu Kai Tang Ming Liu Baoquan Chen
Deep neural network models have achieved remarkable progress in 3D scene
understanding while trained in the closed-set setting and with full labels.
However, the major bottleneck for current 3D recognition approaches is that
they do not have the capacity to recognize any unseen novel classes beyond the
training categories in diverse kinds of real-world applications. In the
meantime, current state-of-the-art 3D scene understanding approaches primarily
require high-quality labels to train neural networks, which merely perform well
in a fully supervised manner. This work presents a generalized and simple
framework for dealing with 3D scene understanding when the labeled scenes are
quite limited. To extract knowledge for novel categories from the pre-trained
vision-language models, we propose a hierarchical feature-aligned pre-training
and knowledge distillation strategy to extract and distill meaningful
information from large-scale vision-language models, which helps benefit the
open-vocabulary scene understanding tasks. To leverage the boundary
information, we propose a novel energy-based loss with boundary awareness
benefiting from the region-level boundary predictions. To encourage latent
instance discrimination and to guarantee efficiency, we propose the
unsupervised region-level semantic contrastive learning scheme for point
clouds, using confident predictions of the neural network to discriminate the
intermediate feature embeddings at multiple stages. Extensive experiments with
both indoor and outdoor scenes demonstrated the effectiveness of our approach
in both data-efficient learning and open-world few-shot learning. All codes,
models, and data are made publicly available at:
https://drive.google.com/drive/folders/1M58V-PtR8DBEwD296zJkNg_m2qq-MTAP?usp=sharing.