Segment Any 3D Object with Language
Seungjun Lee Yuyang Zhao Gim Hee Lee
In this paper, we investigate Open-Vocabulary 3D Instance Segmentation
(OV-3DIS) with free-form language instructions. Earlier works that rely on only
annotated base categories for training suffer from limited generalization to
unseen novel categories. Recent works mitigate poor generalizability to novel
categories by generating class-agnostic masks or projecting generalized masks
from 2D to 3D, but disregard semantic or geometry information, leading to
sub-optimal performance. Instead, generating generalizable but semantic-related
masks directly from 3D point clouds would result in superior outcomes. In this
paper, we introduce Segment any 3D Object with LanguagE (SOLE), which is a
semantic and geometric-aware visual-language learning framework with strong
generalizability by generating semantic-related masks directly from 3D point
clouds. Specifically, we propose a multimodal fusion network to incorporate
multimodal semantics in both backbone and decoder. In addition, to align the 3D
segmentation model with various language instructions and enhance the mask
quality, we introduce three types of multimodal associations as supervision.
Our SOLE outperforms previous methods by a large margin on ScanNetv2,
ScanNet200, and Replica benchmarks, and the results are even close to the
fully-supervised counterpart despite the absence of class annotations in the
training. Furthermore, extensive qualitative results demonstrate the
versatility of our SOLE to language instructions.