SC VALL-E: Style-Controllable Zero-Shot Text to Speech Synthesizer
Daegyeom Kim Seongho Hong Yong-Hoon Choi
Expressive speech synthesis models are trained by adding corpora with diverse
speakers, various emotions, and different speaking styles to the dataset, in
order to control various characteristics of speech and generate the desired
voice. In this paper, we propose a style control (SC) VALL-E model based on the
neural codec language model (called VALL-E), which follows the structure of the
generative pretrained transformer 3 (GPT-3). The proposed SC VALL-E takes input
from text sentences and prompt audio and is designed to generate controllable
speech by not simply mimicking the characteristics of the prompt audio but by
controlling the attributes to produce diverse voices. We identify tokens in the
style embedding matrix of the newly designed style network that represent
attributes such as emotion, speaking rate, pitch, and voice intensity, and
design a model that can control these attributes. To evaluate the performance
of SC VALL-E, we conduct comparative experiments with three representative
expressive speech synthesis models: global style token (GST) Tacotron2,
variational autoencoder (VAE) Tacotron2, and original VALL-E. We measure word
error rate (WER), F0 voiced error (FVE), and F0 gross pitch error (F0GPE) as
evaluation metrics to assess the accuracy of generated sentences. For comparing
the quality of synthesized speech, we measure comparative mean option score
(CMOS) and similarity mean option score (SMOS). To evaluate the style control
ability of the generated speech, we observe the changes in F0 and
mel-spectrogram by modifying the trained tokens. When using prompt audio that
is not present in the training data, SC VALL-E generates a variety of
expressive sounds and demonstrates competitive performance compared to the
existing models. Our implementation, pretrained models, and audio samples are
located on GitHub.