Iterative Shallow Fusion of Backward Language Model for End-to-End
  Speech Recognition
Atsunori Ogawa Takafumi Moriya Naoyuki Kamo Naohiro Tawara Marc Delcroix
We propose a new shallow fusion (SF) method to exploit an external backward
language model (BLM) for end-to-end automatic speech recognition (ASR). The BLM
has complementary characteristics with a forward language model (FLM), and the
effectiveness of their combination has been confirmed by rescoring ASR
hypotheses as post-processing. In the proposed SF, we iteratively apply the BLM
to partial ASR hypotheses in the backward direction (i.e., from the possible
next token to the start symbol) during decoding, substituting the newly
calculated BLM scores for the scores calculated at the last iteration. To
enhance the effectiveness of this iterative SF (ISF), we train a partial
sentence-aware BLM (PBLM) using reversed text data including partial sentences,
considering the framework of ISF. In experiments using an attention-based
encoder-decoder ASR system, we confirmed that ISF using the PBLM shows
comparable performance with SF using the FLM. By performing ISF, early pruning
of prospective hypotheses can be prevented during decoding, and we can obtain a
performance improvement compared to applying the PBLM as post-processing.
Finally, we confirmed that, by combining SF and ISF, further performance
improvement can be obtained thanks to the complementarity of the FLM and PBLM.