A One Stop 3D Target Reconstruction and multilevel Segmentation Method
Jiexiong Xu Weikun Zhao Zhiyan Tang Xiangchao Gan
3D object reconstruction and multilevel segmentation are fundamental to
computer vision research. Existing algorithms usually perform 3D scene
reconstruction and target objects segmentation independently, and the
performance is not fully guaranteed due to the challenge of the 3D
segmentation. Here we propose an open-source one stop 3D target reconstruction
and multilevel segmentation framework (OSTRA), which performs segmentation on
2D images, tracks multiple instances with segmentation labels in the image
sequence, and then reconstructs labelled 3D objects or multiple parts with
Multi-View Stereo (MVS) or RGBD-based 3D reconstruction methods. We extend
object tracking and 3D reconstruction algorithms to support continuous
segmentation labels to leverage the advances in the 2D image segmentation,
especially the Segment-Anything Model (SAM) which uses the pretrained neural
network without additional training for new scenes, for 3D object segmentation.
OSTRA supports most popular 3D object models including point cloud, mesh and
voxel, and achieves high performance for semantic segmentation, instance
segmentation and part segmentation on several 3D datasets. It even surpasses
the manual segmentation in scenes with complex structures and occlusions. Our
method opens up a new avenue for reconstructing 3D targets embedded with rich
multi-scale segmentation information in complex scenes. OSTRA is available from
https://github.com/ganlab/OSTRA.