Fast Sparse View Guided NeRF Update for Object Reconfigurations
Ziqi Lu Jianbo Ye Xiaohan Fei Xiaolong Li Jiawei Mo Ashwin Swaminathan Stefano Soatto
Neural Radiance Field (NeRF), as an implicit 3D scene representation, lacks
inherent ability to accommodate changes made to the initial static scene. If
objects are reconfigured, it is difficult to update the NeRF to reflect the new
state of the scene without time-consuming data re-capturing and NeRF
re-training. To address this limitation, we develop the first update method for
NeRFs to physical changes. Our method takes only sparse new images (e.g. 4) of
the altered scene as extra inputs and update the pre-trained NeRF in around 1
to 2 minutes. Particularly, we develop a pipeline to identify scene changes and
update the NeRF accordingly. Our core idea is the use of a second helper NeRF
to learn the local geometry and appearance changes, which sidesteps the
optimization difficulties in direct NeRF fine-tuning. The interpolation power
of the helper NeRF is the key to accurately reconstruct the un-occluded objects
regions under sparse view supervision. Our method imposes no constraints on
NeRF pre-training, and requires no extra user input or explicit semantic
priors. It is an order of magnitude faster than re-training NeRF from scratch
while maintaining on-par and even superior performance.