Towards Accurate Lip-to-Speech Synthesis in-the-Wild
Sindhu Hegde Rudrabha Mukhopadhyay C. V. Jawahar Vinay Namboodiri
In this paper, we introduce a novel approach to address the task of
synthesizing speech from silent videos of any in-the-wild speaker solely based
on lip movements. The traditional approach of directly generating speech from
lip videos faces the challenge of not being able to learn a robust language
model from speech alone, resulting in unsatisfactory outcomes. To overcome this
issue, we propose incorporating noisy text supervision using a state-of-the-art
lip-to-text network that instills language information into our model. The
noisy text is generated using a pre-trained lip-to-text model, enabling our
approach to work without text annotations during inference. We design a visual
text-to-speech network that utilizes the visual stream to generate accurate
speech, which is in-sync with the silent input video. We perform extensive
experiments and ablation studies, demonstrating our approach's superiority over
the current state-of-the-art methods on various benchmark datasets. Further, we
demonstrate an essential practical application of our method in assistive
technology by generating speech for an ALS patient who has lost the voice but
can make mouth movements. Our demo video, code, and additional details can be
found at
\url{http://cvit.iiit.ac.in/research/projects/cvit-projects/ms-l2s-itw}.