Cross-modal & Cross-domain Learning for Unsupervised LiDAR Semantic
  Segmentation
Yiyang Chen Shanshan Zhao Changxing Ding Liyao Tang Chaoyue Wang Dacheng Tao
In recent years, cross-modal domain adaptation has been studied on the paired
2D image and 3D LiDAR data to ease the labeling costs for 3D LiDAR semantic
segmentation (3DLSS) in the target domain. However, in such a setting the
paired 2D and 3D data in the source domain are still collected with additional
effort. Since the 2D-3D projections can enable the 3D model to learn semantic
information from the 2D counterpart, we ask whether we could further remove the
need of source 3D data and only rely on the source 2D images. To answer it,
this paper studies a new 3DLSS setting where a 2D dataset (source) with
semantic annotations and a paired but unannotated 2D image and 3D LiDAR data
(target) are available. To achieve 3DLSS in this scenario, we propose
Cross-Modal and Cross-Domain Learning (CoMoDaL). Specifically, our CoMoDaL aims
at modeling 1) inter-modal cross-domain distillation between the unpaired
source 2D image and target 3D LiDAR data, and 2) the intra-domain cross-modal
guidance between the target 2D image and 3D LiDAR data pair. In CoMoDaL, we
propose to apply several constraints, such as point-to-pixel and
prototype-to-pixel alignments, to associate the semantics in different
modalities and domains by constructing mixed samples in two modalities. The
experimental results on several datasets show that in the proposed setting, the
developed CoMoDaL can achieve segmentation without the supervision of labeled
LiDAR data. Ablations are also conducted to provide more analysis. Code will be
available publicly.