DiffTalker: Co-driven audio-image diffusion for talking faces via
  intermediate landmarks
Zipeng Qi Xulong Zhang Ning Cheng Jing Xiao Jianzong Wang
Generating realistic talking faces is a complex and widely discussed task
with numerous applications. In this paper, we present DiffTalker, a novel model
designed to generate lifelike talking faces through audio and landmark
co-driving. DiffTalker addresses the challenges associated with directly
applying diffusion models to audio control, which are traditionally trained on
text-image pairs. DiffTalker consists of two agent networks: a
transformer-based landmarks completion network for geometric accuracy and a
diffusion-based face generation network for texture details. Landmarks play a
pivotal role in establishing a seamless connection between the audio and image
domains, facilitating the incorporation of knowledge from pre-trained diffusion
models. This innovative approach efficiently produces articulate-speaking
faces. Experimental results showcase DiffTalker's superior performance in
producing clear and geometrically accurate talking faces, all without the need
for additional alignment between audio and image features.