Analysis of Visual Features for Continuous Lipreading in Spanish
David Gimeno-Gómez Carlos-D. Martínez-Hinarejos
During a conversation, our brain is responsible for combining information
obtained from multiple senses in order to improve our ability to understand the
message we are perceiving. Different studies have shown the importance of
presenting visual information in these situations. Nevertheless, lipreading is
a complex task whose objective is to interpret speech when audio is not
available. By dispensing with a sense as crucial as hearing, it will be
necessary to be aware of the challenge that this lack presents. In this paper,
we propose an analysis of different speech visual features with the intention
of identifying which of them is the best approach to capture the nature of lip
movements for natural Spanish and, in this way, dealing with the automatic
visual speech recognition task. In order to estimate our system, we present an
audiovisual corpus compiled from a subset of the RTVE database, which has been
used in the Albayz\'in evaluations. We employ a traditional system based on
Hidden Markov Models with Gaussian Mixture Models. Results show that, although
the task is difficult, in restricted conditions we obtain recognition results
which determine that using eigenlips in combination with deep features is the
best visual approach.