DiffHarmony: Latent Diffusion Model Meets Image Harmonization
Pengfei Zhou Fangxiang Feng Xiaojie Wang
Image harmonization, which involves adjusting the foreground of a composite
image to attain a unified visual consistency with the background, can be
conceptualized as an image-to-image translation task. Diffusion models have
recently promoted the rapid development of image-to-image translation tasks .
However, training diffusion models from scratch is computationally intensive.
Fine-tuning pre-trained latent diffusion models entails dealing with the
reconstruction error induced by the image compression autoencoder, making it
unsuitable for image generation tasks that involve pixel-level evaluation
metrics. To deal with these issues, in this paper, we first adapt a pre-trained
latent diffusion model to the image harmonization task to generate the
harmonious but potentially blurry initial images. Then we implement two
strategies: utilizing higher-resolution images during inference and
incorporating an additional refinement stage, to further enhance the clarity of
the initially harmonized images. Extensive experiments on iHarmony4 datasets
demonstrate the superiority of our proposed method. The code and model will be
made publicly available at https://github.com/nicecv/DiffHarmony .