Marrying NeRF with Feature Matching for One-step Pose Estimation
Ronghan Chen Yang Cong Yu Ren
Given the image collection of an object, we aim at building a real-time
image-based pose estimation method, which requires neither its CAD model nor
hours of object-specific training. Recent NeRF-based methods provide a
promising solution by directly optimizing the pose from pixel loss between
rendered and target images. However, during inference, they require long
converging time, and suffer from local minima, making them impractical for
real-time robot applications. We aim at solving this problem by marrying image
matching with NeRF. With 2D matches and depth rendered by NeRF, we directly
solve the pose in one step by building 2D-3D correspondences between target and
initial view, thus allowing for real-time prediction. Moreover, to improve the
accuracy of 2D-3D correspondences, we propose a 3D consistent point mining
strategy, which effectively discards unfaithful points reconstruted by NeRF.
Moreover, current NeRF-based methods naively optimizing pixel loss fail at
occluded images. Thus, we further propose a 2D matches based sampling strategy
to preclude the occluded area. Experimental results on representative datasets
prove that our method outperforms state-of-the-art methods, and improves
inference efficiency by 90x, achieving real-time prediction at 6 FPS.