Face2Diffusion for Fast and Editable Face Personalization
Kaede Shiohara Toshihiko Yamasaki
Face personalization aims to insert specific faces, taken from images, into
pretrained text-to-image diffusion models. However, it is still challenging for
previous methods to preserve both the identity similarity and editability due
to overfitting to training samples. In this paper, we propose Face2Diffusion
(F2D) for high-editability face personalization. The core idea behind F2D is
that removing identity-irrelevant information from the training pipeline
prevents the overfitting problem and improves editability of encoded faces. F2D
consists of the following three novel components: 1) Multi-scale identity
encoder provides well-disentangled identity features while keeping the benefits
of multi-scale information, which improves the diversity of camera poses. 2)
Expression guidance disentangles face expressions from identities and improves
the controllability of face expressions. 3) Class-guided denoising
regularization encourages models to learn how faces should be denoised, which
boosts the text-alignment of backgrounds. Extensive experiments on the
FaceForensics++ dataset and diverse prompts demonstrate our method greatly
improves the trade-off between the identity- and text-fidelity compared to
previous state-of-the-art methods.