NeO 360: Neural Fields for Sparse View Synthesis of Outdoor Scenes
Muhammad Zubair Irshad Sergey Zakharov Katherine Liu Vitor Guizilini Thomas Kollar Adrien Gaidon Zsolt Kira Rares Ambrus
Recent implicit neural representations have shown great results for novel
view synthesis. However, existing methods require expensive per-scene
optimization from many views hence limiting their application to real-world
unbounded urban settings where the objects of interest or backgrounds are
observed from very few views. To mitigate this challenge, we introduce a new
approach called NeO 360, Neural fields for sparse view synthesis of outdoor
scenes. NeO 360 is a generalizable method that reconstructs 360{\deg} scenes
from a single or a few posed RGB images. The essence of our approach is in
capturing the distribution of complex real-world outdoor 3D scenes and using a
hybrid image-conditional triplanar representation that can be queried from any
world point. Our representation combines the best of both voxel-based and
bird's-eye-view (BEV) representations and is more effective and expressive than
each. NeO 360's representation allows us to learn from a large collection of
unbounded 3D scenes while offering generalizability to new views and novel
scenes from as few as a single image during inference. We demonstrate our
approach on the proposed challenging 360{\deg} unbounded dataset, called NeRDS
360, and show that NeO 360 outperforms state-of-the-art generalizable methods
for novel view synthesis while also offering editing and composition
capabilities. Project page:
https://zubair-irshad.github.io/projects/neo360.html