SAMPro3D: Locating SAM Prompts in 3D for Zero-Shot Scene Segmentation
Mutian Xu Xingyilang Yin Lingteng Qiu Yang Liu Xin Tong Xiaoguang Han
We introduce SAMPro3D for zero-shot 3D indoor scene segmentation. Given the
3D point cloud and multiple posed 2D frames of 3D scenes, our approach segments
3D scenes by applying the pretrained Segment Anything Model (SAM) to 2D frames.
Our key idea involves locating 3D points in scenes as natural 3D prompts to
align their projected pixel prompts across frames, ensuring frame-consistency
in both pixel prompts and their SAM-predicted masks. Moreover, we suggest
filtering out low-quality 3D prompts based on feedback from all 2D frames, for
enhancing segmentation quality. We also propose to consolidate different 3D
prompts if they are segmenting the same object, bringing a more comprehensive
segmentation. Notably, our method does not require any additional training on
domain-specific data, enabling us to preserve the zero-shot power of SAM.
Extensive qualitative and quantitative results show that our method
consistently achieves higher quality and more diverse segmentation than
previous zero-shot or fully supervised approaches, and in many cases even
surpasses human-level annotations. The project page can be accessed at
https://mutianxu.github.io/sampro3d/.