LEAP: Liberate Sparse-view 3D Modeling from Camera Poses
Hanwen Jiang Zhenyu Jiang Yue Zhao Qixing Huang
Are camera poses necessary for multi-view 3D modeling? Existing approaches
predominantly assume access to accurate camera poses. While this assumption
might hold for dense views, accurately estimating camera poses for sparse views
is often elusive. Our analysis reveals that noisy estimated poses lead to
degraded performance for existing sparse-view 3D modeling methods. To address
this issue, we present LEAP, a novel pose-free approach, therefore challenging
the prevailing notion that camera poses are indispensable. LEAP discards
pose-based operations and learns geometric knowledge from data. LEAP is
equipped with a neural volume, which is shared across scenes and is
parameterized to encode geometry and texture priors. For each incoming scene,
we update the neural volume by aggregating 2D image features in a
feature-similarity-driven manner. The updated neural volume is decoded into the
radiance field, enabling novel view synthesis from any viewpoint. On both
object-centric and scene-level datasets, we show that LEAP significantly
outperforms prior methods when they employ predicted poses from
state-of-the-art pose estimators. Notably, LEAP performs on par with prior
approaches that use ground-truth poses while running $400\times$ faster than
PixelNeRF. We show LEAP generalizes to novel object categories and scenes, and
learns knowledge closely resembles epipolar geometry. Project page:
https://hwjiang1510.github.io/LEAP/