Open-NeRF: Towards Open Vocabulary NeRF Decomposition
Hao Zhang Fang Li Narendra Ahuja
In this paper, we address the challenge of decomposing Neural Radiance Fields
(NeRF) into objects from an open vocabulary, a critical task for object
manipulation in 3D reconstruction and view synthesis. Current techniques for
NeRF decomposition involve a trade-off between the flexibility of processing
open-vocabulary queries and the accuracy of 3D segmentation. We present,
Open-vocabulary Embedded Neural Radiance Fields (Open-NeRF), that leverage
large-scale, off-the-shelf, segmentation models like the Segment Anything Model
(SAM) and introduce an integrate-and-distill paradigm with hierarchical
embeddings to achieve both the flexibility of open-vocabulary querying and 3D
segmentation accuracy. Open-NeRF first utilizes large-scale foundation models
to generate hierarchical 2D mask proposals from varying viewpoints. These
proposals are then aligned via tracking approaches and integrated within the 3D
space and subsequently distilled into the 3D field. This process ensures
consistent recognition and granularity of objects from different viewpoints,
even in challenging scenarios involving occlusion and indistinct features. Our
experimental results show that the proposed Open-NeRF outperforms
state-of-the-art methods such as LERF \cite{lerf} and FFD \cite{ffd} in
open-vocabulary scenarios. Open-NeRF offers a promising solution to NeRF
decomposition, guided by open-vocabulary queries, enabling novel applications
in robotics and vision-language interaction in open-world 3D scenes.