Towards cross-language prosody transfer for dialog
Jonathan E. Avila Nigel G. Ward
Speech-to-speech translation systems today do not adequately support use for
dialog purposes. In particular, nuances of speaker intent and stance can be
lost due to improper prosody transfer. We present an exploration of what needs
to be done to overcome this. First, we developed a data collection protocol in
which bilingual speakers re-enact utterances from an earlier conversation in
their other language, and used this to collect an English-Spanish corpus, so
far comprising 1871 matched utterance pairs. Second, we developed a simple
prosodic dissimilarity metric based on Euclidean distance over a broad set of
prosodic features. We then used these to investigate cross-language prosodic
differences, measure the likely utility of three simple baseline models, and
identify phenomena which will require more powerful modeling. Our findings
should inform future research on cross-language prosody and the design of
speech-to-speech translation systems capable of effective prosody transfer.