FreePIH: Training-Free Painterly Image Harmonization with Diffusion
  Model
Ruibin Li Jingcai Guo Song Guo Qihua Zhou Jie Zhang
This paper provides an efficient training-free painterly image harmonization
(PIH) method, dubbed FreePIH, that leverages only a pre-trained diffusion model
to achieve state-of-the-art harmonization results. Unlike existing methods that
require either training auxiliary networks or fine-tuning a large pre-trained
backbone, or both, to harmonize a foreground object with a painterly-style
background image, our FreePIH tames the denoising process as a plug-in module
for foreground image style transfer. Specifically, we find that the very last
few steps of the denoising (i.e., generation) process strongly correspond to
the stylistic information of images, and based on this, we propose to augment
the latent features of both the foreground and background images with Gaussians
for a direct denoising-based harmonization. To guarantee the fidelity of the
harmonized image, we make use of multi-scale features to enforce the
consistency of the content and stability of the foreground objects in the
latent space, and meanwhile, aligning both fore-/back-grounds with the same
style. Moreover, to accommodate the generation with more structural and
textural details, we further integrate text prompts to attend to the latent
features, hence improving the generation quality. Quantitative and qualitative
evaluations on COCO and LAION 5B datasets demonstrate that our method can
surpass representative baselines by large margins.