Noise-robust zero-shot text-to-speech synthesis conditioned on
  self-supervised speech-representation model with adapters
Kenichi Fujita Hiroshi Sato Takanori Ashihara Hiroki Kanagawa Marc Delcroix Takafumi Moriya Yusuke Ijima
The zero-shot text-to-speech (TTS) method, based on speaker embeddings
extracted from reference speech using self-supervised learning (SSL) speech
representations, can reproduce speaker characteristics very accurately.
However, this approach suffers from degradation in speech synthesis quality
when the reference speech contains noise. In this paper, we propose a
noise-robust zero-shot TTS method. We incorporated adapters into the SSL model,
which we fine-tuned with the TTS model using noisy reference speech. In
addition, to further improve performance, we adopted a speech enhancement (SE)
front-end. With these improvements, our proposed SSL-based zero-shot TTS
achieved high-quality speech synthesis with noisy reference speech. Through the
objective and subjective evaluations, we confirmed that the proposed method is
highly robust to noise in reference speech, and effectively works in
combination with SE.