It's Never Too Late: Fusing Acoustic Information into Large Language
  Models for Automatic Speech Recognition
Chen Chen Ruizhe Li Yuchen Hu Sabato Marco Siniscalchi Pin-Yu Chen Ensiong Chng Chao-Han Huck Yang
Recent studies have successfully shown that large language models (LLMs) can
be successfully used for generative error correction (GER) on top of the
automatic speech recognition (ASR) output. Specifically, an LLM is utilized to
carry out a direct mapping from the N-best hypotheses list generated by an ASR
system to the predicted output transcription. However, despite its
effectiveness, GER introduces extra data uncertainty since the LLM is trained
without taking into account acoustic information available in the speech
signal. In this work, we aim to overcome such a limitation by infusing acoustic
information before generating the predicted transcription through a novel late
fusion solution termed Uncertainty-Aware Dynamic Fusion (UADF). UADF is a
multimodal fusion approach implemented into an auto-regressive decoding process
and works in two stages: (i) It first analyzes and calibrates the token-level
LLM decision, and (ii) it then dynamically assimilates the information from the
acoustic modality. Experimental evidence collected from various ASR tasks shows
that UADF surpasses existing fusion mechanisms in several ways. It yields
significant improvements in word error rate (WER) while mitigating data
uncertainty issues in LLM and addressing the poor generalization relied with
sole modality during fusion. We also demonstrate that UADF seamlessly adapts to
audio-visual speech recognition.