Multilingual Speech-to-Speech Translation into Multiple Target Languages
Hongyu Gong Ning Dong Sravya Popuri Vedanuj Goswami Ann Lee Juan Pino
Speech-to-speech translation (S2ST) enables spoken communication between
people talking in different languages. Despite a few studies on multilingual
S2ST, their focus is the multilinguality on the source side, i.e., the
translation from multiple source languages to one target language. We present
the first work on multilingual S2ST supporting multiple target languages.
Leveraging recent advance in direct S2ST with speech-to-unit and vocoder, we
equip these key components with multilingual capability. Speech-to-masked-unit
(S2MU) is the multilingual extension of S2U, which applies masking to units
which don't belong to the given target language to reduce the language
interference. We also propose multilingual vocoder which is trained with
language embedding and the auxiliary loss of language identification. On
benchmark translation testsets, our proposed multilingual model shows superior
performance than bilingual models in the translation from English into $16$
target languages.