Omni-Recon: Towards General-Purpose Neural Radiance Fields for Versatile
  3D Applications
Yonggan Fu Huaizhi Qu Zhifan Ye Chaojian Li Kevin Zhao Yingyan Lin
Recent breakthroughs in Neural Radiance Fields (NeRFs) have sparked
significant demand for their integration into real-world 3D applications.
However, the varied functionalities required by different 3D applications often
necessitate diverse NeRF models with various pipelines, leading to tedious NeRF
training for each target task and cumbersome trial-and-error experiments.
Drawing inspiration from the generalization capability and adaptability of
emerging foundation models, our work aims to develop one general-purpose NeRF
for handling diverse 3D tasks. We achieve this by proposing a framework called
Omni-Recon, which is capable of (1) generalizable 3D reconstruction and
zero-shot multitask scene understanding, and (2) adaptability to diverse
downstream 3D applications such as real-time rendering and scene editing. Our
key insight is that an image-based rendering pipeline, with accurate geometry
and appearance estimation, can lift 2D image features into their 3D
counterparts, thus extending widely explored 2D tasks to the 3D world in a
generalizable manner. Specifically, our Omni-Recon features a general-purpose
NeRF model using image-based rendering with two decoupled branches: one complex
transformer-based branch that progressively fuses geometry and appearance
features for accurate geometry estimation, and one lightweight branch for
predicting blending weights of source views. This design achieves
state-of-the-art (SOTA) generalizable 3D surface reconstruction quality with
blending weights reusable across diverse tasks for zero-shot multitask scene
understanding. In addition, it can enable real-time rendering after baking the
complex geometry branch into meshes, swift adaptation to achieve SOTA
generalizable 3D understanding performance, and seamless integration with 2D
diffusion models for text-guided 3D editing.