MAD Speech: Measures of Acoustic Diversity of Speech
Matthieu Futeral Andrea Agostinelli Marco Tagliasacchi Neil Zeghidour Eugene Kharitonov
Generative spoken language models produce speech in a wide range of voices,
prosody, and recording conditions, seemingly approaching the diversity of
natural speech. However, the extent to which generated speech is acoustically
diverse remains unclear due to a lack of appropriate metrics. We address this
gap by developing lightweight metrics of acoustic diversity, which we
collectively refer to as MAD Speech. We focus on measuring five facets of
acoustic diversity: voice, gender, emotion, accent, and background noise. We
construct the metrics as a composition of specialized, per-facet embedding
models and an aggregation function that measures diversity within the embedding
space. Next, we build a series of datasets with a priori known diversity
preferences for each facet. Using these datasets, we demonstrate that our
proposed metrics achieve a stronger agreement with the ground-truth diversity
than baselines. Finally, we showcase the applicability of our proposed metrics
across several real-life evaluation scenarios. MAD Speech will be made publicly
accessible.