3DFusion, A real-time 3D object reconstruction pipeline based on
  streamed instance segmented data
Xi Sun Derek Jacoby Yvonne Coady
This paper presents a real-time segmentation and reconstruction system that
utilizes RGB-D images to generate accurate and detailed individual 3D models of
objects within a captured scene. Leveraging state-of-the-art instance
segmentation techniques, the system performs pixel-level segmentation on RGB-D
data, effectively separating foreground objects from the background. The
segmented objects are then reconstructed into distinct 3D models in a
high-performance computation platform. The real-time 3D modelling can be
applied across various domains, including augmented/virtual reality, interior
design, urban planning, road assistance, security systems, and more. To achieve
real-time performance, the paper proposes a method that effectively samples
consecutive frames to reduce network load while ensuring reconstruction
quality. Additionally, a multi-process SLAM pipeline is adopted for parallel 3D
reconstruction, enabling efficient cutting of the clustering objects into
individuals. This system employs the industry-leading framework YOLO for
instance segmentation. To improve YOLO's performance and accuracy,
modifications were made to resolve duplicated or false detection of similar
objects, ensuring the reconstructed models align with the targets. Overall,
this work establishes a robust real-time system with a significant enhancement
for object segmentation and reconstruction in the indoor environment. It can
potentially be extended to the outdoor scenario, opening up numerous
opportunities for real-world applications.