NeRF-VO: Real-Time Sparse Visual Odometry with Neural Radiance Fields
Jens Naumann Binbin Xu Stefan Leutenegger Xingxing Zuo
We introduce a novel monocular visual odometry (VO) system, NeRF-VO, that
integrates learning-based sparse visual odometry for low-latency camera
tracking and a neural radiance scene representation for sophisticated dense
reconstruction and novel view synthesis. Our system initializes camera poses
using sparse visual odometry and obtains view-dependent dense geometry priors
from a monocular depth prediction network. We harmonize the scale of poses and
dense geometry, treating them as supervisory cues to train a neural implicit
scene representation. NeRF-VO demonstrates exceptional performance in both
photometric and geometric fidelity of the scene representation by jointly
optimizing a sliding window of keyframed poses and the underlying dense
geometry, which is accomplished through training the radiance field with volume
rendering. We surpass state-of-the-art methods in pose estimation accuracy,
novel view synthesis fidelity, and dense reconstruction quality across a
variety of synthetic and real-world datasets, while achieving a higher camera
tracking frequency and consuming less GPU memory.