TranSentence: Speech-to-speech Translation via Language-agnostic
  Sentence-level Speech Encoding without Language-parallel Data
Seung-Bin Kim Sang-Hoon Lee Seong-Whan Lee
Although there has been significant advancement in the field of
speech-to-speech translation, conventional models still require
language-parallel speech data between the source and target languages for
training. In this paper, we introduce TranSentence, a novel speech-to-speech
translation without language-parallel speech data. To achieve this, we first
adopt a language-agnostic sentence-level speech encoding that captures the
semantic information of speech, irrespective of language. We then train our
model to generate speech based on the encoded embedding obtained from a
language-agnostic sentence-level speech encoder that is pre-trained with
various languages. With this method, despite training exclusively on the target
language's monolingual data, we can generate target language speech in the
inference stage using language-agnostic speech embedding from the source
language speech. Furthermore, we extend TranSentence to multilingual
speech-to-speech translation. The experimental results demonstrate that
TranSentence is superior to other models.