This paper proposes a neural end-to-end text-to-speech (TTS) model which can control latent attributes in the generated speech that are rarely annotated in the training data, such as speaking style, accent, background noise, and recording conditions. The model is formulated as a conditional generative model with two levels of hierarchical latent variables. The first level is a categorical variable, which represents attribute groups (e.g. clean/noisy) and provides interpretability. The second level, conditioned on the first, is a multivariate Gaussian variable, which characterizes specific attribute configurations (e.g. noise level, speaking rate) and enables disentangled fine-grained control over these attributes. This amounts to using a Gaussian mixture model (GMM) for the latent distribution. Extensive evaluation demonstrates its ability to control the aforementioned attributes. In particular, it is capable of consistently synthesizing high-quality clean speech regardless of the quality of the training data for the target speaker. Recent development of neural sequence-to-sequence TTS models has shown promising results in generating high fidelity speech without the need of handcrafted linguistic features BID30 BID37 BID2 . These models rely heavily on a encoderdecoder neural network structure BID31 BID5 that maps a text sequence to a sequence of speech frames. Extensions to these models have shown that attributes such as speaker identity can be controlled by conditioning the decoder on additional attribute labels BID3 .There are many speech attributes aside from speaker identity that are difficult to annotate, such as speaking style, prosody, recording channel, and noise levels. ; model such latent attributes through conditional auto-encoding, by extending the decoder inputs to include a vector inferred from the target speech which aims to capture the residual attributes that are not specified by other input streams, in addition to text and a speaker label. These models have shown convincing results in synthesizing speech that resembles the prosody or the noise conditions of the reference speech, which may not have the same text or speaker identity as the target speech.Nevertheless, the presence of multiple latent attributes is common in crowdsourced data such as BID26 , in which prosody, speaker, and noise conditions all vary simultaneously. Using such data , simply copying the latent attributes from a reference is insufficient if one desires to synthesize speech that mimics the prosody of the reference, but is in the same noise condition as another. If the latent representation were disentangled, these generating factors could be controlled independently. Furthermore, it is can useful to construct a systematic method for synthesizing speech with random latent attributes, which would facilitate data augmentation BID33 BID12 BID8 by generating diverse examples. These properties were not explicitly addressed in the previous studies, which model variation of a single latent attribute.Motivated by the applications of sampling, inferring, and independently controlling individual attributes, we build off of and extend Tacotron 2 to model two separate latent spaces: one for labeled (i.e. related to speaker identity) and another for unlabeled attributes. Each latent variable is modeled in a variational autoencoding BID22 ) framework using Gaussian mixture priors. The resulting latent spaces (1) learn disentangled attribute representations, where each dimension controls a different generating factor; (2) discover a set of interpretable clusters, each of which corresponds to a representative mode in the training data (e.g., one cluster for clean speech and another for noisy speech); and (3) provide a systematic sampling mechanism from the learned prior. The proposed model is extensively evaluated on four datasets with subjective and objective quantitative metrics, as well as comprehensive qualitative studies. Experiments confirm that the proposed model is capable of controlling speaker, noise, and style independently, even when variation of all attributes is present but unannotated in the train set.Our main contributions are as follows:• We propose a principled probabilistic hierarchical generative model, which improves (1) sampling stability and disentangled attribute control compared to e.g. the GST model of , and (2) interpretability and quality compared to e.g. BID0 .• The model formulation explicitly factors the latent encoding by using two mixture distributions to separately model supervised speaker attributes and latent attributes in a disentangled fashion. This makes it straightforward to condition the model output on speaker and latent encodings inferred from different reference utterances.• To the best of our knowledge, this work is the first to train a high-quality controllable textto-speech system on real found data containing significant variation in recording condition, speaker identity, as well as prosody and style. Previous results on similar data focused on speaker modeling , and did not explicitly address modeling of prosody and background noise. Leveraging disentangled speaker and latent attribute encodings, the proposed model is capable of inferring the speaker attribute representation from a noisy utterance spoken by a previously unseen speaker, and using it to synthesize high-quality clean speech that approximates the voice of that speaker. We describe GMVAE-Tacotron, a TTS model which learns an interpretable and disentangled latent representation to enable fine-grained control of latent attributes and provides a systematic sampling scheme for them. If speaker labels are available, we demonstrate an extension of the model that learns a continuous space that captures speaker attributes, along with an inference model which enables one-shot learning of speaker attributes from unseen reference utterances.The proposed model was extensively evaluated on tasks spanning a wide range of signal variation. We demonstrated that it can independently control many latent attributes, and is able to cluster them without supervision. In particular, we verified using both subjective and objective tests that the model could synthesize high-quality clean speech for a target speaker even if the quality of data for that speaker does not meet high standard. These experimental results demonstrated the effectiveness of the model for training high-quality controllable TTS systems on large scale training data with rich styles by learning to factorize and independently control latent attributes underlying the speech signal.
Building a TTS model with Gaussian Mixture VAEs enables fine-grained control of speaking style, noise condition, and more. Describes the conditioned GAN model to generate speaker conditioned Mel spectra by augmenting the z-space corresponding to the identification This paper proposes a two layer latent variable model to obtain disentangled latent representation, thus facilitating fine-grained control over various attributes This paper proposes a model that can control non-annotated attributes such as speaking style, accent, background noise, etc.