Robust Acoustic and Semantic Contextual Biasing in Neural Transducers
  for Speech Recognition
Xuandi Fu Kanthashree Mysore Sathyendra Ankur Gandhe Jing Liu Grant P. Strimel Ross McGowan Athanasios Mouchtaris
Attention-based contextual biasing approaches have shown significant
improvements in the recognition of generic and/or personal rare-words in
End-to-End Automatic Speech Recognition (E2E ASR) systems like neural
transducers. These approaches employ cross-attention to bias the model towards
specific contextual entities injected as bias-phrases to the model. Prior
approaches typically relied on subword encoders for encoding the bias phrases.
However, subword tokenizations are coarse and fail to capture granular
pronunciation information which is crucial for biasing based on acoustic
similarity. In this work, we propose to use lightweight character
representations to encode fine-grained pronunciation features to improve
contextual biasing guided by acoustic similarity between the audio and the
contextual entities (termed acoustic biasing). We further integrate pretrained
neural language model (NLM) based encoders to encode the utterance's semantic
context along with contextual entities to perform biasing informed by the
utterance's semantic context (termed semantic biasing). Experiments using a
Conformer Transducer model on the Librispeech dataset show a 4.62% - 9.26%
relative WER improvement on different biasing list sizes over the baseline
contextual model when incorporating our proposed acoustic and semantic biasing
approach. On a large-scale in-house dataset, we observe 7.91% relative WER
improvement compared to our baseline model. On tail utterances, the
improvements are even more pronounced with 36.80% and 23.40% relative WER
improvements on Librispeech rare words and an in-house testset respectively.