SERF: Fine-Grained Interactive 3D Segmentation and Editing with Radiance
  Fields
Kaichen Zhou Lanqing Hong Enze Xie Yongxin Yang Zhenguo Li Wei Zhang
Although significant progress has been made in the field of 2D-based
interactive editing, fine-grained 3D-based interactive editing remains
relatively unexplored. This limitation can be attributed to two main
challenges: the lack of an efficient 3D representation robust to different
modifications and the absence of an effective 3D interactive segmentation
method. In this paper, we introduce a novel fine-grained interactive 3D
segmentation and editing algorithm with radiance fields, which we refer to as
SERF. Our method entails creating a neural mesh representation by integrating
multi-view algorithms with pre-trained 2D models. Building upon this
representation, we introduce a novel surface rendering technique that preserves
local information and is robust to deformation. Moreover, this representation
forms the basis for achieving accurate and interactive 3D segmentation without
requiring 3D supervision. Harnessing this representation facilitates a range of
interactive 3D editing operations, encompassing tasks such as interactive
geometry editing and texture painting. Extensive experiments and visualization
examples of editing on both real and synthetic data demonstrate the superiority
of our method on representation quality and editing ability.