Improving Fairness and Robustness in End-to-End Speech Recognition
  through unsupervised clustering
Irina-Elena Veliche Pascale Fung
The challenge of fairness arises when Automatic Speech Recognition (ASR)
systems do not perform equally well for all sub-groups of the population. In
the past few years there have been many improvements in overall speech
recognition quality, but without any particular focus on advancing Equality and
Equity for all user groups for whom systems do not perform well. ASR fairness
is therefore also a robustness issue. Meanwhile, data privacy also takes
priority in production systems. In this paper, we present a privacy preserving
approach to improve fairness and robustness of end-to-end ASR without using
metadata, zip codes, or even speaker or utterance embeddings directly in
training. We extract utterance level embeddings using a speaker ID model
trained on a public dataset, which we then use in an unsupervised fashion to
create acoustic clusters. We use cluster IDs instead of speaker utterance
embeddings as extra features during model training, which shows improvements
for all demographic groups and in particular for different accents.