A Novel Interpretable and Generalizable Re-synchronization Model for
  Cued Speech based on a Multi-Cuer Corpus
Lufei Gao Shan Huang Li Liu
Cued Speech (CS) is a multi-modal visual coding system combining lip reading
with several hand cues at the phonetic level to make the spoken language
visible to the hearing impaired. Previous studies solved asynchronous problems
between lip and hand movements by a cuer\footnote{The people who perform Cued
Speech are called the cuer.}-dependent piecewise linear model for English and
French CS. In this work, we innovatively propose three statistical measure on
the lip stream to build an interpretable and generalizable model for predicting
hand preceding time (HPT), which achieves cuer-independent by a proper
normalization. Particularly, we build the first Mandarin CS corpus comprising
annotated videos from five speakers including three normal and two hearing
impaired individuals. Consequently, we show that the hand preceding phenomenon
exists in Mandarin CS production with significant differences between normal
and hearing impaired people. Extensive experiments demonstrate that our model
outperforms the baseline and the previous state-of-the-art methods.