XPhoneBERT: A Pre-trained Multilingual Model for Phoneme Representations
  for Text-to-Speech
Linh The Nguyen Thinh Pham Dat Quoc Nguyen
We present XPhoneBERT, the first multilingual model pre-trained to learn
phoneme representations for the downstream text-to-speech (TTS) task. Our
XPhoneBERT has the same model architecture as BERT-base, trained using the
RoBERTa pre-training approach on 330M phoneme-level sentences from nearly 100
languages and locales. Experimental results show that employing XPhoneBERT as
an input phoneme encoder significantly boosts the performance of a strong
neural TTS model in terms of naturalness and prosody and also helps produce
fairly high-quality speech with limited training data. We publicly release our
pre-trained XPhoneBERT with the hope that it would facilitate future research
and downstream TTS applications for multiple languages. Our XPhoneBERT model is
available at https://github.com/VinAIResearch/XPhoneBERT