A Comparative Study on E-Branchformer vs Conformer in Speech
  Recognition, Translation, and Understanding Tasks
Yifan Peng Kwangyoun Kim Felix Wu Brian Yan Siddhant Arora William Chen Jiyang Tang Suwon Shon Prashant Sridhar Shinji Watanabe
Conformer, a convolution-augmented Transformer variant, has become the de
facto encoder architecture for speech processing due to its superior
performance in various tasks, including automatic speech recognition (ASR),
speech translation (ST) and spoken language understanding (SLU). Recently, a
new encoder called E-Branchformer has outperformed Conformer in the LibriSpeech
ASR benchmark, making it promising for more general speech applications. This
work compares E-Branchformer and Conformer through extensive experiments using
different types of end-to-end sequence-to-sequence models. Results demonstrate
that E-Branchformer achieves comparable or better performance than Conformer in
almost all evaluation sets across 15 ASR, 2 ST, and 3 SLU benchmarks, while
being more stable during training. We will release our training configurations
and pre-trained models for reproducibility, which can benefit the speech
community.