Rethinking Speech Recognition with A Multimodal Perspective via Acoustic
  and Semantic Cooperative Decoding
Tian-Hao Zhang Hai-Bo Qin Zhi-Hao Lai Song-Lu Chen Qi Liu Feng Chen Xinyuan Qian Xu-Cheng Yin
Attention-based encoder-decoder (AED) models have shown impressive
performance in ASR. However, most existing AED methods neglect to
simultaneously leverage both acoustic and semantic features in decoder, which
is crucial for generating more accurate and informative semantic states. In
this paper, we propose an Acoustic and Semantic Cooperative Decoder (ASCD) for
ASR. In particular, unlike vanilla decoders that process acoustic and semantic
features in two separate stages, ASCD integrates them cooperatively. To prevent
information leakage during training, we design a Causal Multimodal Mask.
Moreover, a variant Semi-ASCD is proposed to balance accuracy and computational
cost. Our proposal is evaluated on the publicly available AISHELL-1 and
aidatatang_200zh datasets using Transformer, Conformer, and Branchformer as
encoders, respectively. The experimental results show that ASCD significantly
improves the performance by leveraging both the acoustic and semantic
information cooperatively.