Houston we have a Divergence: A Subgroup Performance Analysis of ASR
  Models
Alkis Koudounas Flavio Giobergia
The Fearless Steps APOLLO Community Resource provides unparalleled
opportunities to explore the potential of multi-speaker team communications
from NASA Apollo missions. This study focuses on discovering the
characteristics that make Apollo recordings more or less intelligible to
Automatic Speech Recognition (ASR) methods. We extract, for each audio
recording, interpretable metadata on recordings (signal-to-noise ratio,
spectral flatness, presence of pauses, sentence duration), transcript (number
of words spoken, speaking rate), or known a priori (speaker). We identify
subgroups of audio recordings based on combinations of these metadata and
compute each subgroup's performance (e.g., Word Error Rate) and the difference
in performance (''divergence'') w.r.t the overall population. We then apply the
Whisper model in different sizes, trained on English-only or multilingual
datasets, in zero-shot or after fine-tuning. We conduct several analyses to (i)
automatically identify and describe the most problematic subgroups for a given
model, (ii) examine the impact of fine-tuning w.r.t. zero-shot at the subgroup
level, (iii) understand the effect of model size on subgroup performance, and
(iv) analyze if multilingual models are more sensitive than monolingual to
subgroup performance disparities. The insights enhance our understanding of
subgroup-specific performance variations, paving the way for advancements in
optimizing ASR systems for Earth-to-space communications.