Improved Factorized Neural Transducer Model For text-only Domain
  Adaptation
Junzhe Liu Jianwei Yu Xie Chen
End-to-end models, such as the neural Transducer, have been successful in
integrating acoustic and linguistic information jointly to achieve excellent
recognition performance. However, adapting these models with text-only data is
challenging. Factorized neural Transducer (FNT) aims to address this issue by
introducing a separate vocabulary decoder to predict the vocabulary, which can
effectively perform traditional text data adaptation. Nonetheless, this
approach has limitations in fusing acoustic and language information
seamlessly. Moreover, a degradation in word error rate (WER) on the general
test sets was also observed, leading to doubts about its overall performance.
In response to this challenge, we present an improved factorized neural
Transducer (IFNT) model structure designed to comprehensively integrate
acoustic and language information while enabling effective text adaptation. We
evaluate the performance of our proposed methods through in-domain experiments
on GigaSpeech and out-of-domain experiments adapting to EuroParl, TED-LIUM, and
Medical datasets. After text-only adaptation, IFNT yields 7.9% to 28.5%
relative WER improvements over the standard neural Transducer with shallow
fusion, and relative WER reductions ranging from 1.6% to 8.2% on the three test
sets compared to the FNT model.