Multi-Modal Automatic Prosody Annotation with Contrastive Pretraining of
  SSWP
Jinzuomu Zhong Yang Li Hui Huang Jie Liu Zhiba Su Jing Guo Benlai Tang Fengjie Zhu
In the realm of expressive Text-to-Speech (TTS), explicit prosodic boundaries
significantly advance the naturalness and controllability of synthesized
speech. While human prosody annotation contributes a lot to the performance, it
is a labor-intensive and time-consuming process, often resulting in
inconsistent outcomes. Despite the availability of extensive supervised data,
the current benchmark model still faces performance setbacks. To address this
issue, a two-stage automatic annotation pipeline is novelly proposed in this
paper. Specifically, in the first stage, we propose contrastive text-speech
pretraining of Speech-Silence and Word-Punctuation (SSWP) pairs. The
pretraining procedure hammers at enhancing the prosodic space extracted from
joint text-speech space. In the second stage, we build a multi-modal prosody
annotator, which consists of pretrained encoders, a straightforward yet
effective text-speech feature fusion scheme, and a sequence classifier.
Extensive experiments conclusively demonstrate that our proposed method excels
at automatically generating prosody annotation and achieves state-of-the-art
(SOTA) performance. Furthermore, our novel model has exhibited remarkable
resilience when tested with varying amounts of data.