Face Swap via Diffusion Model
Feifei Wang
This technical report presents a diffusion model based framework for face
swapping between two portrait images. The basic framework consists of three
components, i.e., IP-Adapter, ControlNet, and Stable Diffusion's inpainting
pipeline, for face feature encoding, multi-conditional generation, and face
inpainting respectively. Besides, I introduce facial guidance optimization and
CodeFormer based blending to further improve the generation quality.
  Specifically, we engage a recent light-weighted customization method (i.e.,
DreamBooth-LoRA), to guarantee the identity consistency by 1) using a rare
identifier "sks" to represent the source identity, and 2) injecting the image
features of source portrait into each cross-attention layer like the text
features. Then I resort to the strong inpainting ability of Stable Diffusion,
and utilize canny image and face detection annotation of the target portrait as
the conditions, to guide ContorlNet's generation and align source portrait with
the target portrait. To further correct face alignment, we add the facial
guidance loss to optimize the text embedding during the sample generation.