Cooperative Dual Attention for Audio-Visual Speech Enhancement with
  Facial Cues
Feixiang Wang Shuang Yang Shiguang Shan Xilin Chen
In this work, we focus on leveraging facial cues beyond the lip region for
robust Audio-Visual Speech Enhancement (AVSE). The facial region, encompassing
the lip region, reflects additional speech-related attributes such as gender,
skin color, nationality, etc., which contribute to the effectiveness of AVSE.
However, static and dynamic speech-unrelated attributes also exist, causing
appearance changes during speech. To address these challenges, we propose a
Dual Attention Cooperative Framework, DualAVSE, to ignore speech-unrelated
information, capture speech-related information with facial cues, and
dynamically integrate it with the audio signal for AVSE. Specifically, we
introduce a spatial attention-based visual encoder to capture and enhance
visual speech information beyond the lip region, incorporating global facial
context and automatically ignoring speech-unrelated information for robust
visual feature extraction. Additionally, a dynamic visual feature fusion
strategy is introduced by integrating a temporal-dimensional self-attention
module, enabling the model to robustly handle facial variations. The acoustic
noise in the speaking process is variable, impacting audio quality. Therefore,
a dynamic fusion strategy for both audio and visual features is introduced to
address this issue. By integrating cooperative dual attention in the visual
encoder and audio-visual fusion strategy, our model effectively extracts
beneficial speech information from both audio and visual cues for AVSE.
Thorough analysis and comparison on different datasets, including normal and
challenging cases with unreliable or absent visual information, consistently
show our model outperforming existing methods across multiple metrics.