Code-Mixed Text to Speech Synthesis under Low-Resource Constraints
Raviraj Joshi Nikesh Garera
Text-to-speech (TTS) systems are an important component in voice-based
e-commerce applications. These applications include end-to-end voice assistant
and customer experience (CX) voice bot. Code-mixed TTS is also relevant in
these applications since the product names are commonly described in English
while the surrounding text is in a regional language. In this work, we describe
our approaches for production quality code-mixed Hindi-English TTS systems
built for e-commerce applications. We propose a data-oriented approach by
utilizing monolingual data sets in individual languages. We leverage a
transliteration model to convert the Roman text into a common Devanagari script
and then combine both datasets for training. We show that such single script
bi-lingual training without any code-mixing works well for pure code-mixed test
sets. We further present an exhaustive evaluation of single-speaker adaptation
and multi-speaker training with Tacotron2 + Waveglow setup to show that the
former approach works better. These approaches are also coupled with transfer
learning and decoder-only fine-tuning to improve performance. We compare these
approaches with the Google TTS and report a positive CMOS score of 0.02 with
the proposed transfer learning approach. We also perform low-resource voice
adaptation experiments to show that a new voice can be onboarded with just 3
hrs of data. This highlights the importance of our pre-trained models in
resource-constrained settings. This subjective evaluation is performed on a
large number of out-of-domain pure code-mixed sentences to demonstrate the high
quality of the systems.