GO-NeRF: Generating Virtual Objects in Neural Radiance Fields
Peng Dai Feitong Tan Xin Yu Yinda Zhang Xiaojuan Qi
Despite advances in 3D generation, the direct creation of 3D objects within
an existing 3D scene represented as NeRF remains underexplored. This process
requires not only high-quality 3D object generation but also seamless
composition of the generated 3D content into the existing NeRF. To this end, we
propose a new method, GO-NeRF, capable of utilizing scene context for
high-quality and harmonious 3D object generation within an existing NeRF. Our
method employs a compositional rendering formulation that allows the generated
3D objects to be seamlessly composited into the scene utilizing learned
3D-aware opacity maps without introducing unintended scene modification.
Moreover, we also develop tailored optimization objectives and training
strategies to enhance the model's ability to exploit scene context and mitigate
artifacts, such as floaters, originating from 3D object generation within a
scene. Extensive experiments on both feed-forward and $360^o$ scenes show the
superior performance of our proposed GO-NeRF in generating objects harmoniously
composited with surrounding scenes and synthesizing high-quality novel view
images. Project page at {\url{https://daipengwa.github.io/GO-NeRF/}.