DisControlFace: Disentangled Control for Personalized Facial Image
  Editing
Haozhe Jia Yan Li Hengfei Cui Di Xu Changpeng Yang Yuwang Wang Tao Yu
In this work, we focus on exploring explicit fine-grained control of
generative facial image editing, all while generating faithful and consistent
personalized facial appearances. We identify the key challenge of this task as
the exploration of disentangled conditional control in the generation process,
and accordingly propose a novel diffusion-based framework, named
DisControlFace, comprising two decoupled components. Specifically, we leverage
an off-the-shelf diffusion reconstruction model as the backbone and freeze its
pre-trained weights, which helps to reduce identity shift and recover
editing-unrelated details of the input image. Furthermore, we construct a
parallel control network that is compatible with the reconstruction backbone to
generate spatial control conditions based on estimated explicit face
parameters. Finally, we further reformulate the training pipeline into a
masked-autoencoding form to effectively achieve disentangled training of our
DisControlFace. Our DisControlNet can perform robust editing on any facial
image through training on large-scale 2D in-the-wild portraits and also
supports low-cost fine-tuning with few additional images to further learn
diverse personalized priors of a specific person. Extensive experiments
demonstrate that DisControlFace can generate realistic facial images
corresponding to various face control conditions, while significantly improving
the preservation of the personalized facial details.