Utilizing Neural Transducers for Two-Stage Text-to-Speech via Semantic
  Token Prediction
Minchan Kim Myeonghun Jeong Byoung Jin Choi Semin Kim Joun Yeop Lee Nam Soo Kim
We propose a novel text-to-speech (TTS) framework centered around a neural
transducer. Our approach divides the whole TTS pipeline into semantic-level
sequence-to-sequence (seq2seq) modeling and fine-grained acoustic modeling
stages, utilizing discrete semantic tokens obtained from wav2vec2.0 embeddings.
For a robust and efficient alignment modeling, we employ a neural transducer
named token transducer for the semantic token prediction, benefiting from its
hard monotonic alignment constraints. Subsequently, a non-autoregressive (NAR)
speech generator efficiently synthesizes waveforms from these semantic tokens.
Additionally, a reference speech controls temporal dynamics and acoustic
conditions at each stage. This decoupled framework reduces the training
complexity of TTS while allowing each stage to focus on semantic and acoustic
modeling. Our experimental results on zero-shot adaptive TTS demonstrate that
our model surpasses the baseline in terms of speech quality and speaker
similarity, both objectively and subjectively. We also delve into the inference
speed and prosody control capabilities of our approach, highlighting the
potential of neural transducers in TTS frameworks.