Maximizing Data Efficiency for Cross-Lingual TTS Adaptation by
  Self-Supervised Representation Mixing and Embedding Initialization
Wei-Ping Huang Sung-Feng Huang Hung-yi Lee
This paper presents an effective transfer learning framework for language
adaptation in text-to-speech systems, with a focus on achieving language
adaptation using minimal labeled and unlabeled data. While many works focus on
reducing the usage of labeled data, very few consider minimizing the usage of
unlabeled data. By utilizing self-supervised features in the pretraining stage,
replacing the noisy portion of pseudo labels with these features during
fine-tuning, and incorporating an embedding initialization trick, our method
leverages more information from unlabeled data compared to conventional
approaches. Experimental results show that our framework is able to synthesize
intelligible speech in unseen languages with only 4 utterances of labeled data
and 15 minutes of unlabeled data. Our methodology continues to surpass
conventional techniques, even when a greater volume of data is accessible.
These findings highlight the potential of our data-efficient language
adaptation framework.