Fine-grained Appearance Transfer with Diffusion Models
Yuteng Ye Guanwen Li Hang Zhou Cai Jiale Junqing Yu Yawei Luo Zikai Song Qilong Xing Youjia Zhang Wei Yang
Image-to-image translation (I2I), and particularly its subfield of appearance
transfer, which seeks to alter the visual appearance between images while
maintaining structural coherence, presents formidable challenges. Despite
significant advancements brought by diffusion models, achieving fine-grained
transfer remains complex, particularly in terms of retaining detailed
structural elements and ensuring information fidelity. This paper proposes an
innovative framework designed to surmount these challenges by integrating
various aspects of semantic matching, appearance transfer, and latent
deviation. A pivotal aspect of our approach is the strategic use of the
predicted $x_0$ space by diffusion models within the latent space of diffusion
processes. This is identified as a crucial element for the precise and natural
transfer of fine-grained details. Our framework exploits this space to
accomplish semantic alignment between source and target images, facilitating
mask-wise appearance transfer for improved feature acquisition. A significant
advancement of our method is the seamless integration of these features into
the latent space, enabling more nuanced latent deviations without necessitating
extensive model retraining or fine-tuning. The effectiveness of our approach is
demonstrated through extensive experiments, which showcase its ability to
adeptly handle fine-grained appearance transfers across a wide range of
categories and domains. We provide our code at
https://github.com/babahui/Fine-grained-Appearance-Transfer