Understanding and Bridging the Modality Gap for Speech Translation
Qingkai Fang Yang Feng
How to achieve better end-to-end speech translation (ST) by leveraging (text)
machine translation (MT) data? Among various existing techniques, multi-task
learning is one of the effective ways to share knowledge between ST and MT in
which additional MT data can help to learn source-to-target mapping. However,
due to the differences between speech and text, there is always a gap between
ST and MT. In this paper, we first aim to understand this modality gap from the
target-side representation differences, and link the modality gap to another
well-known problem in neural machine translation: exposure bias. We find that
the modality gap is relatively small during training except for some difficult
cases, but keeps increasing during inference due to the cascading effect. To
address these problems, we propose the Cross-modal Regularization with
Scheduled Sampling (Cress) method. Specifically, we regularize the output
predictions of ST and MT, whose target-side contexts are derived by sampling
between ground truth words and self-generated words with a varying probability.
Furthermore, we introduce token-level adaptive training which assigns different
training weights to target tokens to handle difficult cases with large modality
gaps. Experiments and analysis show that our approach effectively bridges the
modality gap, and achieves promising results in all eight directions of the
MuST-C dataset.