Accelerator-Aware Training for Transducer-Based Speech Recognition
Suhaila M. Shakiah Rupak Vignesh Swaminathan Hieu Duy Nguyen Raviteja Chinta Tariq Afzal Nathan Susanj Athanasios Mouchtaris Grant P. Strimel Ariya Rastrow
Machine learning model weights and activations are represented in
full-precision during training. This leads to performance degradation in
runtime when deployed on neural network accelerator (NNA) chips, which leverage
highly parallelized fixed-point arithmetic to improve runtime memory and
latency. In this work, we replicate the NNA operators during the training
phase, accounting for the degradation due to low-precision inference on the NNA
in back-propagation. Our proposed method efficiently emulates NNA operations,
thus foregoing the need to transfer quantization error-prone data to the
Central Processing Unit (CPU), ultimately reducing the user perceived latency
(UPL). We apply our approach to Recurrent Neural Network-Transducer (RNN-T), an
attractive architecture for on-device streaming speech recognition tasks. We
train and evaluate models on 270K hours of English data and show a 5-7%
improvement in engine latency while saving up to 10% relative degradation in
WER.