Memory-augmented conformer for improved end-to-end long-form ASR
Carlos Carvalho Alberto Abad
Conformers have recently been proposed as a promising modelling approach for
automatic speech recognition (ASR), outperforming recurrent neural
network-based approaches and transformers. Nevertheless, in general, the
performance of these end-to-end models, especially attention-based models, is
particularly degraded in the case of long utterances. To address this
limitation, we propose adding a fully-differentiable memory-augmented neural
network between the encoder and decoder of a conformer. This external memory
can enrich the generalization for longer utterances since it allows the system
to store and retrieve more information recurrently. Notably, we explore the
neural Turing machine (NTM) that results in our proposed Conformer-NTM model
architecture for ASR. Experimental results using Librispeech train-clean-100
and train-960 sets show that the proposed system outperforms the baseline
conformer without memory for long utterances.