DiffFAE: Advancing High-fidelity One-shot Facial Appearance Editing with
  Space-sensitive Customization and Semantic Preservation
Qilin Wang Jiangning Zhang Chengming Xu Weijian Cao Ying Tai Yue Han Yanhao Ge Hong Gu Chengjie Wang Yanwei Fu
Facial Appearance Editing (FAE) aims to modify physical attributes, such as
pose, expression and lighting, of human facial images while preserving
attributes like identity and background, showing great importance in
photograph. In spite of the great progress in this area, current researches
generally meet three challenges: low generation fidelity, poor attribute
preservation, and inefficient inference. To overcome above challenges, this
paper presents DiffFAE, a one-stage and highly-efficient diffusion-based
framework tailored for high-fidelity FAE. For high-fidelity query attributes
transfer, we adopt Space-sensitive Physical Customization (SPC), which ensures
the fidelity and generalization ability by utilizing rendering texture derived
from 3D Morphable Model (3DMM). In order to preserve source attributes, we
introduce the Region-responsive Semantic Composition (RSC). This module is
guided to learn decoupled source-regarding features, thereby better preserving
the identity and alleviating artifacts from non-facial attributes such as hair,
clothes, and background. We further introduce a consistency regularization for
our pipeline to enhance editing controllability by leveraging prior knowledge
in the attention matrices of diffusion model. Extensive experiments demonstrate
the superiority of DiffFAE over existing methods, achieving state-of-the-art
performance in facial appearance editing.