InFusion: Inpainting 3D Gaussians via Learning Depth Completion from
  Diffusion Prior
Zhiheng Liu Hao Ouyang Qiuyu Wang Ka Leong Cheng Jie Xiao Kai Zhu Nan Xue Yu Liu Yujun Shen Yang Cao
3D Gaussians have recently emerged as an efficient representation for novel
view synthesis. This work studies its editability with a particular focus on
the inpainting task, which aims to supplement an incomplete set of 3D Gaussians
with additional points for visually harmonious rendering. Compared to 2D
inpainting, the crux of inpainting 3D Gaussians is to figure out the
rendering-relevant properties of the introduced points, whose optimization
largely benefits from their initial 3D positions. To this end, we propose to
guide the point initialization with an image-conditioned depth completion
model, which learns to directly restore the depth map based on the observed
image. Such a design allows our model to fill in depth values at an aligned
scale with the original depth, and also to harness strong generalizability from
largescale diffusion prior. Thanks to the more accurate depth completion, our
approach, dubbed InFusion, surpasses existing alternatives with sufficiently
better fidelity and efficiency under various complex scenarios. We further
demonstrate the effectiveness of InFusion with several practical applications,
such as inpainting with user-specific texture or with novel object insertion.