Speech Translation with Foundation Models and Optimal Transport: UPC at
  IWSLT23
Ioannis Tsiamas Gerard I. Gállego José A. R. Fonollosa Marta R. Costa-jussà
This paper describes the submission of the UPC Machine Translation group to
the IWSLT 2023 Offline Speech Translation task. Our Speech Translation systems
utilize foundation models for speech (wav2vec 2.0) and text (mBART50). We
incorporate a Siamese pretraining step of the speech and text encoders with CTC
and Optimal Transport, to adapt the speech representations to the space of the
text model, thus maximizing transfer learning from MT. After this pretraining,
we fine-tune our system end-to-end on ST, with Cross Entropy and Knowledge
Distillation. Apart from the available ST corpora, we create synthetic data
with SegAugment to better adapt our models to the custom segmentations of the
IWSLT test sets. Our best single model obtains 31.2 BLEU points on MuST-C
tst-COMMON, 29.8 points on IWLST.tst2020 and 33.4 points on the newly released
IWSLT.ACLdev2023.