Improving speech translation by fusing speech and text
Wenbiao Yin Zhicheng Liu Chengqi Zhao Tao Wang Jian Tong Rong Ye
In speech translation, leveraging multimodal data to improve model
performance and address limitations of individual modalities has shown
significant effectiveness. In this paper, we harness the complementary
strengths of speech and text, which are disparate modalities. We observe three
levels of modality gap between them, denoted by Modal input representation,
Modal semantic, and Modal hidden states. To tackle these gaps, we propose
\textbf{F}use-\textbf{S}peech-\textbf{T}ext (\textbf{FST}), a cross-modal model
which supports three distinct input modalities for translation: speech, text,
and fused speech-text. We leverage multiple techniques for cross-modal
alignment and conduct a comprehensive analysis to assess its impact on speech
translation, machine translation, and fused speech-text translation. We
evaluate FST on MuST-C, GigaST, and newstest benchmark. Experiments show that
the proposed FST achieves an average 34.0 BLEU on MuST-C
En$\rightarrow$De/Es/Fr (vs SOTA +1.1 BLEU). Further experiments demonstrate
that FST does not degrade on MT task, as observed in prior works. Instead, it
yields an average improvement of 3.2 BLEU over the pre-trained MT model.