Reference-based Painterly Inpainting via Diffusion: Crossing the Wild
  Reference Domain Gap
Dejia Xu Xingqian Xu Wenyan Cong Humphrey Shi Zhangyang Wang
Have you ever imagined how it would look if we placed new objects into
paintings? For example, what would it look like if we placed a basketball into
Claude Monet's ``Water Lilies, Evening Effect''? We propose Reference-based
Painterly Inpainting, a novel task that crosses the wild reference domain gap
and implants novel objects into artworks. Although previous works have examined
reference-based inpainting, they are not designed for large domain
discrepancies between the target and the reference, such as inpainting an
artistic image using a photorealistic reference. This paper proposes a novel
diffusion framework, dubbed RefPaint, to ``inpaint more wildly'' by taking such
references with large domain gaps. Built with an image-conditioned diffusion
model, we introduce a ladder-side branch and a masked fusion mechanism to work
with the inpainting mask. By decomposing the CLIP image embeddings at inference
time, one can manipulate the strength of semantic and style information with
ease. Experiments demonstrate that our proposed RefPaint framework produces
significantly better results than existing methods. Our method enables creative
painterly image inpainting with reference objects that would otherwise be
difficult to achieve. Project page: https://vita-group.github.io/RefPaint/