DiffProsody: Diffusion-based Latent Prosody Generation for Expressive
  Speech Synthesis with Prosody Conditional Adversarial Training
Hyung-Seok Oh Sang-Hoon Lee Seong-Whan Lee
Expressive text-to-speech systems have undergone significant advancements
owing to prosody modeling, but conventional methods can still be improved.
Traditional approaches have relied on the autoregressive method to predict the
quantized prosody vector; however, it suffers from the issues of long-term
dependency and slow inference. This study proposes a novel approach called
DiffProsody in which expressive speech is synthesized using a diffusion-based
latent prosody generator and prosody conditional adversarial training. Our
findings confirm the effectiveness of our prosody generator in generating a
prosody vector. Furthermore, our prosody conditional discriminator
significantly improves the quality of the generated speech by accurately
emulating prosody. We use denoising diffusion generative adversarial networks
to improve the prosody generation speed. Consequently, DiffProsody is capable
of generating prosody 16 times faster than the conventional diffusion model.
The superior performance of our proposed method has been demonstrated via
experiments.