Quantifying the redundancy between prosody and text
Lukas Wolf Tiago Pimentel Evelina Fedorenko Ryan Cotterell Alex Warstadt Ethan Wilcox Tamar Regev
Prosody -- the suprasegmental component of speech, including pitch, loudness,
and tempo -- carries critical aspects of meaning. However, the relationship
between the information conveyed by prosody vs. by the words themselves remains
poorly understood. We use large language models (LLMs) to estimate how much
information is redundant between prosody and the words themselves. Using a
large spoken corpus of English audiobooks, we extract prosodic features aligned
to individual words and test how well they can be predicted from LLM
embeddings, compared to non-contextual word embeddings. We find a high degree
of redundancy between the information carried by the words and prosodic
information across several prosodic features, including intensity, duration,
pauses, and pitch contours. Furthermore, a word's prosodic information is
redundant with both the word itself and the context preceding as well as
following it. Still, we observe that prosodic features can not be fully
predicted from text, suggesting that prosody carries information above and
beyond the words. Along with this paper, we release a general-purpose data
processing pipeline for quantifying the relationship between linguistic
information and extra-linguistic features.