Hearing Lips in Noise: Universal Viseme-Phoneme Mapping and Transfer for
  Robust Audio-Visual Speech Recognition
Yuchen Hu Ruizhe Li Chen Chen Chengwei Qin Qiushi Zhu Eng Siong Chng
Audio-visual speech recognition (AVSR) provides a promising solution to
ameliorate the noise-robustness of audio-only speech recognition with visual
information. However, most existing efforts still focus on audio modality to
improve robustness considering its dominance in AVSR task, with noise
adaptation techniques such as front-end denoise processing. Though effective,
these methods are usually faced with two practical challenges: 1) lack of
sufficient labeled noisy audio-visual training data in some real-world
scenarios and 2) less optimal model generality to unseen testing noises. In
this work, we investigate the noise-invariant visual modality to strengthen
robustness of AVSR, which can adapt to any testing noises while without
dependence on noisy training data, a.k.a., unsupervised noise adaptation.
Inspired by human perception mechanism, we propose a universal viseme-phoneme
mapping (UniVPM) approach to implement modality transfer, which can restore
clean audio from visual signals to enable speech recognition under any noisy
conditions. Extensive experiments on public benchmarks LRS3 and LRS2 show that
our approach achieves the state-of-the-art under various noisy as well as clean
conditions. In addition, we also outperform previous state-of-the-arts on
visual speech recognition task.