Cross-lingual Knowledge Distillation via Flow-based Voice Conversion for
  Robust Polyglot Text-To-Speech
Dariusz Piotrowski Renard Korzeniowski Alessio Falai Sebastian Cygert Kamil Pokora Georgi Tinchev Ziyao Zhang Kayoko Yanagisawa
In this work, we introduce a framework for cross-lingual speech synthesis,
which involves an upstream Voice Conversion (VC) model and a downstream
Text-To-Speech (TTS) model. The proposed framework consists of 4 stages. In the
first two stages, we use a VC model to convert utterances in the target locale
to the voice of the target speaker. In the third stage, the converted data is
combined with the linguistic features and durations from recordings in the
target language, which are then used to train a single-speaker acoustic model.
Finally, the last stage entails the training of a locale-independent vocoder.
Our evaluations show that the proposed paradigm outperforms state-of-the-art
approaches which are based on training a large multilingual TTS model. In
addition, our experiments demonstrate the robustness of our approach with
different model architectures, languages, speakers and amounts of data.
Moreover, our solution is especially beneficial in low-resource settings.