Refining Segmentation On-the-Fly: An Interactive Framework for Point
  Cloud Semantic Segmentation
Peng Zhang Ting Wu Jinsheng Sun Weiqing Li Zhiyong Su
Existing interactive point cloud segmentation approaches primarily focus on
the object segmentation, which aim to determine which points belong to the
object of interest guided by user interactions. This paper concentrates on an
unexplored yet meaningful task, i.e., interactive point cloud semantic
segmentation, which assigns high-quality semantic labels to all points in a
scene with user corrective clicks. Concretely, we presents the first
interactive framework for point cloud semantic segmentation, named InterPCSeg,
which seamlessly integrates with off-the-shelf semantic segmentation networks
without offline re-training, enabling it to run in an on-the-fly manner. To
achieve online refinement, we treat user interactions as sparse training
examples during the test-time. To address the instability caused by the sparse
supervision, we design a stabilization energy to regulate the test-time
training process. For objective and reproducible evaluation, we develop an
interaction simulation scheme tailored for the interactive point cloud semantic
segmentation task. We evaluate our framework on the S3DIS and ScanNet datasets
with off-the-shelf segmentation networks, incorporating interactions from both
the proposed interaction simulator and real users. Quantitative and qualitative
experimental results demonstrate the efficacy of our framework in refining the
semantic segmentation results with user interactions. The source code will be
publicly available.