SALTTS: Leveraging Self-Supervised Speech Representations for improved
  Text-to-Speech Synthesis
Ramanan Sivaguru Vasista Sai Lodagala S Umesh
While FastSpeech2 aims to integrate aspects of speech such as pitch, energy,
and duration as conditional inputs, it still leaves scope for richer
representations. As a part of this work, we leverage representations from
various Self-Supervised Learning (SSL) models to enhance the quality of the
synthesized speech. In particular, we pass the FastSpeech2 encoder's
length-regulated outputs through a series of encoder layers with the objective
of reconstructing the SSL representations. In the SALTTS-parallel
implementation, the representations from this second encoder are used for an
auxiliary reconstruction loss with the SSL features. The SALTTS-cascade
implementation, however, passes these representations through the decoder in
addition to having the reconstruction loss. The richness of speech
characteristics from the SSL features reflects in the output speech quality,
with the objective and subjective evaluation measures of the proposed approach
outperforming the baseline FastSpeech2.