Bridging the Gaps of Both Modality and Language: Synchronous Bilingual
  CTC for Speech Translation and Speech Recognition
Chen Xu Xiaoqian Liu Erfeng He Yuhao Zhang Qianqian Dong Tong Xiao Jingbo Zhu Dapeng Man Wu Yang
In this study, we present synchronous bilingual Connectionist Temporal
Classification (CTC), an innovative framework that leverages dual CTC to bridge
the gaps of both modality and language in the speech translation (ST) task.
Utilizing transcript and translation as concurrent objectives for CTC, our
model bridges the gap between audio and text as well as between source and
target languages. Building upon the recent advances in CTC application, we
develop an enhanced variant, BiL-CTC+, that establishes new state-of-the-art
performances on the MuST-C ST benchmarks under resource-constrained scenarios.
Intriguingly, our method also yields significant improvements in speech
recognition performance, revealing the effect of cross-lingual learning on
transcription and demonstrating its broad applicability. The source code is
available at https://github.com/xuchennlp/S2T.