Inpaint4DNeRF: Promptable Spatio-Temporal NeRF Inpainting with
  Generative Diffusion Models
Han Jiang Haosen Sun Ruoxuan Li Chi-Keung Tang Yu-Wing Tai
Current Neural Radiance Fields (NeRF) can generate photorealistic novel
views. For editing 3D scenes represented by NeRF, with the advent of generative
models, this paper proposes Inpaint4DNeRF to capitalize on state-of-the-art
stable diffusion models (e.g., ControlNet) for direct generation of the
underlying completed background content, regardless of static or dynamic. The
key advantages of this generative approach for NeRF inpainting are twofold.
First, after rough mask propagation, to complete or fill in previously occluded
content, we can individually generate a small subset of completed images with
plausible content, called seed images, from which simple 3D geometry proxies
can be derived. Second and the remaining problem is thus 3D multiview
consistency among all completed images, now guided by the seed images and their
3D proxies. Without other bells and whistles, our generative Inpaint4DNeRF
baseline framework is general which can be readily extended to 4D dynamic
NeRFs, where temporal consistency can be naturally handled in a similar way as
our multiview consistency.