Towards Spontaneous Style Modeling with Semi-supervised Pre-training for
  Conversational Text-to-Speech Synthesis
Weiqin Li Shun Lei Qiaochu Huang Yixuan Zhou Zhiyong Wu Shiyin Kang Helen Meng
The spontaneous behavior that often occurs in conversations makes speech more
human-like compared to reading-style. However, synthesizing spontaneous-style
speech is challenging due to the lack of high-quality spontaneous datasets and
the high cost of labeling spontaneous behavior. In this paper, we propose a
semi-supervised pre-training method to increase the amount of spontaneous-style
speech and spontaneous behavioral labels. In the process of semi-supervised
learning, both text and speech information are considered for detecting
spontaneous behaviors labels in speech. Moreover, a linguistic-aware encoder is
used to model the relationship between each sentence in the conversation.
Experimental results indicate that our proposed method achieves superior
expressive speech synthesis performance with the ability to model spontaneous
behavior in spontaneous-style speech and predict reasonable spontaneous
behavior from text.