Cross-Attention Fusion of Visual and Geometric Features for Large
  Vocabulary Arabic Lipreading
Samar Daou Ahmed Rekik Achraf Ben-Hamadou Abdelaziz Kallel
Lipreading involves using visual data to recognize spoken words by analyzing
the movements of the lips and surrounding area. It is a hot research topic with
many potential applications, such as human-machine interaction and enhancing
audio speech recognition. Recent deep-learning based works aim to integrate
visual features extracted from the mouth region with landmark points on the lip
contours. However, employing a simple combination method such as concatenation
may not be the most effective approach to get the optimal feature vector. To
address this challenge, firstly, we propose a cross-attention fusion-based
approach for large lexicon Arabic vocabulary to predict spoken words in videos.
Our method leverages the power of cross-attention networks to efficiently
integrate visual and geometric features computed on the mouth region. Secondly,
we introduce the first large-scale Lip Reading in the Wild for Arabic (LRW-AR)
dataset containing 20,000 videos for 100-word classes, uttered by 36 speakers.
The experimental results obtained on LRW-AR and ArabicVisual databases showed
the effectiveness and robustness of the proposed approach in recognizing Arabic
words. Our work provides insights into the feasibility and effectiveness of
applying lipreading techniques to the Arabic language, opening doors for
further research in this field. Link to the project page:
https://crns-smartvision.github.io/lrwar