Blank-regularized CTC for Frame Skipping in Neural Transducer
Yifan Yang Xiaoyu Yang Liyong Guo Zengwei Yao Wei Kang Fangjun Kuang Long Lin Xie Chen Daniel Povey
Neural Transducer and connectionist temporal classification (CTC) are popular
end-to-end automatic speech recognition systems. Due to their frame-synchronous
design, blank symbols are introduced to address the length mismatch between
acoustic frames and output tokens, which might bring redundant computation.
Previous studies managed to accelerate the training and inference of neural
Transducers by discarding frames based on the blank symbols predicted by a
co-trained CTC. However, there is no guarantee that the co-trained CTC can
maximize the ratio of blank symbols. This paper proposes two novel
regularization methods to explicitly encourage more blanks by constraining the
self-loop of non-blank symbols in the CTC. It is interesting to find that the
frame reduction ratio of the neural Transducer can approach the theoretical
boundary. Experiments on LibriSpeech corpus show that our proposed method
accelerates the inference of neural Transducer by 4 times without sacrificing
performance. Our work is open-sourced and publicly available
https://github.com/k2-fsa/icefall.