Speech Robust Bench: A Robustness Benchmark For Speech Recognition
Muhammad A. Shah David Solans Noguero Mikko A. Heikkila Nicolas Kourtellis
As Automatic Speech Recognition (ASR) models become ever more pervasive, it
is important to ensure that they make reliable predictions under corruptions
present in the physical and digital world. We propose Speech Robust Bench
(SRB), a comprehensive benchmark for evaluating the robustness of ASR models to
diverse corruptions. SRB is composed of 69 input perturbations which are
intended to simulate various corruptions that ASR models may encounter in the
physical and digital world. We use SRB to evaluate the robustness of several
state-of-the-art ASR models and observe that model size and certain modeling
choices such as discrete representations, and self-training appear to be
conducive to robustness. We extend this analysis to measure the robustness of
ASR models on data from various demographic subgroups, namely English and
Spanish speakers, and males and females, and observed noticeable disparities in
the model's robustness across subgroups. We believe that SRB will facilitate
future research towards robust ASR models, by making it easier to conduct
comprehensive and comparable robustness evaluations.