Learning Multilingual Expressive Speech Representation for Prosody
  Prediction without Parallel Data
Jarod Duret Titouan Parcollet Yannick Estève
We propose a method for speech-to-speech emotionpreserving translation that
operates at the level of discrete speech units. Our approach relies on the use
of multilingual emotion embedding that can capture affective information in a
language-independent manner. We show that this embedding can be used to predict
the pitch and duration of speech units in a target language, allowing us to
resynthesize the source speech signal with the same emotional content. We
evaluate our approach to English and French speech signals and show that it
outperforms a baseline method that does not use emotional information,
including when the emotion embedding is extracted from a different language.
Even if this preliminary study does not address directly the machine
translation issue, our results demonstrate the effectiveness of our approach
for cross-lingual emotion preservation in the context of speech resynthesis.