Learning Unified Decompositional and Compositional NeRF for Editable
  Novel View Synthesis
Yuxin Wang Wayne Wu Dan Xu
Implicit neural representations have shown powerful capacity in modeling
real-world 3D scenes, offering superior performance in novel view synthesis. In
this paper, we target a more challenging scenario, i.e., joint scene novel view
synthesis and editing based on implicit neural scene representations.
State-of-the-art methods in this direction typically consider building separate
networks for these two tasks (i.e., view synthesis and editing). Thus, the
modeling of interactions and correlations between these two tasks is very
limited, which, however, is critical for learning high-quality scene
representations. To tackle this problem, in this paper, we propose a unified
Neural Radiance Field (NeRF) framework to effectively perform joint scene
decomposition and composition for modeling real-world scenes. The decomposition
aims at learning disentangled 3D representations of different objects and the
background, allowing for scene editing, while scene composition models an
entire scene representation for novel view synthesis. Specifically, with a
two-stage NeRF framework, we learn a coarse stage for predicting a global
radiance field as guidance for point sampling, and in the second fine-grained
stage, we perform scene decomposition by a novel one-hot object radiance field
regularization module and a pseudo supervision via inpainting to handle
ambiguous background regions occluded by objects. The decomposed object-level
radiance fields are further composed by using activations from the
decomposition module. Extensive quantitative and qualitative results show the
effectiveness of our method for scene decomposition and composition,
outperforming state-of-the-art methods for both novel-view synthesis and
editing tasks.