Acoustic Model Fusion for End-to-end Speech Recognition
Zhihong Lei Mingbin Xu Shiyi Han Leo Liu Zhen Huang Tim Ng Yuanyuan Zhang Ernest Pusateri Mirko Hannemann Yaqiao Deng Man-Hung Siu
Recent advances in deep learning and automatic speech recognition (ASR) have
enabled the end-to-end (E2E) ASR system and boosted the accuracy to a new
level. The E2E systems implicitly model all conventional ASR components, such
as the acoustic model (AM) and the language model (LM), in a single network
trained on audio-text pairs. Despite this simpler system architecture, fusing a
separate LM, trained exclusively on text corpora, into the E2E system has
proven to be beneficial. However, the application of LM fusion presents certain
drawbacks, such as its inability to address the domain mismatch issue inherent
to the internal AM. Drawing inspiration from the concept of LM fusion, we
propose the integration of an external AM into the E2E system to better address
the domain mismatch. By implementing this novel approach, we have achieved a
significant reduction in the word error rate, with an impressive drop of up to
14.3% across varied test sets. We also discovered that this AM fusion approach
is particularly beneficial in enhancing named entity recognition.