Realistic Speech-to-Face Generation with Speech-Conditioned Latent
  Diffusion Model with Face Prior
Jinting Wang Li Liu Jun Wang Hei Victor Cheng
Speech-to-face generation is an intriguing area of research that focuses on
generating realistic facial images based on a speaker's audio speech. However,
state-of-the-art methods employing GAN-based architectures lack stability and
cannot generate realistic face images. To fill this gap, we propose a novel
speech-to-face generation framework, which leverages a Speech-Conditioned
Latent Diffusion Model, called SCLDM. To the best of our knowledge, this is the
first work to harness the exceptional modeling capabilities of diffusion models
for speech-to-face generation. Preserving the shared identity information
between speech and face is crucial in generating realistic results. Therefore,
we employ contrastive pre-training for both the speech encoder and the face
encoder. This pre-training strategy facilitates effective alignment between the
attributes of speech, such as age and gender, and the corresponding facial
characteristics in the face images. Furthermore, we tackle the challenge posed
by excessive diversity in the synthesis process caused by the diffusion model.
To overcome this challenge, we introduce the concept of residuals by
integrating a statistical face prior to the diffusion process. This addition
helps to eliminate the shared component across the faces and enhances the
subtle variations captured by the speech condition. Extensive quantitative,
qualitative, and user study experiments demonstrate that our method can produce
more realistic face images while preserving the identity of the speaker better
than state-of-the-art methods. Highlighting the notable enhancements, our
method demonstrates significant gains in all metrics on the AVSpeech dataset
and Voxceleb dataset, particularly noteworthy are the improvements of 32.17 and
32.72 on the cosine distance metric for the two datasets, respectively.