OmniSeg3D: Omniversal 3D Segmentation via Hierarchical Contrastive
  Learning
Haiyang Ying Yixuan Yin Jinzhi Zhang Fan Wang Tao Yu Ruqi Huang Lu Fang
Towards holistic understanding of 3D scenes, a general 3D segmentation method
is needed that can segment diverse objects without restrictions on object
quantity or categories, while also reflecting the inherent hierarchical
structure. To achieve this, we propose OmniSeg3D, an omniversal segmentation
method aims for segmenting anything in 3D all at once. The key insight is to
lift multi-view inconsistent 2D segmentations into a consistent 3D feature
field through a hierarchical contrastive learning framework, which is
accomplished by two steps. Firstly, we design a novel hierarchical
representation based on category-agnostic 2D segmentations to model the
multi-level relationship among pixels. Secondly, image features rendered from
the 3D feature field are clustered at different levels, which can be further
drawn closer or pushed apart according to the hierarchical relationship between
different levels. In tackling the challenges posed by inconsistent 2D
segmentations, this framework yields a global consistent 3D feature field,
which further enables hierarchical segmentation, multi-object selection, and
global discretization. Extensive experiments demonstrate the effectiveness of
our method on high-quality 3D segmentation and accurate hierarchical structure
understanding. A graphical user interface further facilitates flexible
interaction for omniversal 3D segmentation.