Inpaint3D: 3D Scene Content Generation using 2D Inpainting Diffusion
Kira Prabhu Jane Wu Lynn Tsai Peter Hedman Dan B Goldman Ben Poole Michael Broxton
This paper presents a novel approach to inpainting 3D regions of a scene,
given masked multi-view images, by distilling a 2D diffusion model into a
learned 3D scene representation (e.g. a NeRF). Unlike 3D generative methods
that explicitly condition the diffusion model on camera pose or multi-view
information, our diffusion model is conditioned only on a single masked 2D
image. Nevertheless, we show that this 2D diffusion model can still serve as a
generative prior in a 3D multi-view reconstruction problem where we optimize a
NeRF using a combination of score distillation sampling and NeRF reconstruction
losses. Predicted depth is used as additional supervision to encourage accurate
geometry. We compare our approach to 3D inpainting methods that focus on object
removal. Because our method can generate content to fill any 3D masked region,
we additionally demonstrate 3D object completion, 3D object replacement, and 3D
scene completion.