Pushing the Limits of Zero-shot End-to-End Speech Translation
Ioannis Tsiamas Gerard I. Gállego José A. R. Fonollosa Marta R. Costa-jussà
Data scarcity and the modality gap between the speech and text modalities are
two major obstacles of end-to-end Speech Translation (ST) systems, thus
hindering their performance. Prior work has attempted to mitigate these
challenges by leveraging external MT data and optimizing distance metrics that
bring closer the speech-text representations. However, achieving competitive
results typically requires some ST data. For this reason, we introduce
ZeroSwot, a method for zero-shot ST that bridges the modality gap without any
paired ST data. Leveraging a novel CTC compression and Optimal Transport, we
train a speech encoder using only ASR data, to align with the representation
space of a massively multilingual MT model. The speech encoder seamlessly
integrates with the MT model at inference, enabling direct translation from
speech to text, across all languages supported by the MT model. Our experiments
show that we can effectively close the modality gap without ST data, while our
results on MuST-C and CoVoST demonstrate our method's superiority over not only
previous zero-shot models, but also supervised ones, achieving state-of-the-art
results.