Compact Speech Translation Models via Discrete Speech Units Pretraining
Tsz Kin Lam Alexandra Birch Barry Haddow
Using Self-Supervised Learning (SSL) as model initialization is now common to
obtain strong results in Speech Translation (ST). However, they also impose a
large memory footprint, hindering on-device deployment. In this paper, we
leverage the SSL models by pretraining smaller models on their Discrete Speech
Units (DSU). We pretrain encoder-decoder models on 1) Filterbank-to-DSU and 2)
DSU-to-Translation data, and take the encoder from 1) and the decoder from 2)
to initialise a new model, finetuning this on limited speech-translation data.
The final model becomes compact by using the DSU pretraining to distil the
knowledge of the SSL model. Our method has several benefits over using DSU as
model inputs, such as shorter inference pipeline and robustness over (DSU)
tokenization. In contrast to ASR pretraining, it does not require transcripts,
making it applicable to low-resource settings. Evaluation on CoVoST-2 X-En
shows that our method is >$0.5$ BLEU better than a ST model that directly
finetune the SSL model, given only half the model size, and on a par with ASR
pretraining.