Conversational Speech Recognition by Learning Audio-textual Cross-modal
  Contextual Representation
Kun Wei Bei Li Hang Lv Quan Lu Ning Jiang Lei Xie
Automatic Speech Recognition (ASR) in conversational settings presents unique
challenges, including extracting relevant contextual information from previous
conversational turns. Due to irrelevant content, error propagation, and
redundancy, existing methods struggle to extract longer and more effective
contexts. To address this issue, we introduce a novel Conversational ASR
system, extending the Conformer encoder-decoder model with cross-modal
conversational representation. Our approach leverages a cross-modal extractor
that combines pre-trained speech and text models through a specialized encoder
and a modal-level mask input. This enables the extraction of richer historical
speech context without explicit error propagation. We also incorporate
conditional latent variational modules to learn conversational level attributes
such as role preference and topic coherence. By introducing both cross-modal
and conversational representations into the decoder, our model retains context
over longer sentences without information loss, achieving relative accuracy
improvements of 8.8% and 23% on Mandarin conversation datasets HKUST and
MagicData-RAMC, respectively, compared to the standard Conformer model.