Landmark-Guided Cross-Speaker Lip Reading with Mutual Information
  Regularization
Linzhi Wu Xingyu Zhang Yakun Zhang Changyan Zheng Tiejun Liu Liang Xie Ye Yan Erwei Yin
Lip reading, the process of interpreting silent speech from visual lip
movements, has gained rising attention for its wide range of realistic
applications. Deep learning approaches greatly improve current lip reading
systems. However, lip reading in cross-speaker scenarios where the speaker
identity changes, poses a challenging problem due to inter-speaker variability.
A well-trained lip reading system may perform poorly when handling a brand new
speaker. To learn a speaker-robust lip reading model, a key insight is to
reduce visual variations across speakers, avoiding the model overfitting to
specific speakers. In this work, in view of both input visual clues and latent
representations based on a hybrid CTC/attention architecture, we propose to
exploit the lip landmark-guided fine-grained visual clues instead of
frequently-used mouth-cropped images as input features, diminishing
speaker-specific appearance characteristics. Furthermore, a max-min mutual
information regularization approach is proposed to capture speaker-insensitive
latent representations. Experimental evaluations on public lip reading datasets
demonstrate the effectiveness of the proposed approach under the intra-speaker
and inter-speaker conditions.