A Deep Dive into the Disparity of Word Error Rates Across Thousands of
  NPTEL MOOC Videos
Anand Kumar Rai Siddharth D Jaiswal Animesh Mukherjee
Automatic speech recognition (ASR) systems are designed to transcribe spoken
language into written text and find utility in a variety of applications
including voice assistants and transcription services. However, it has been
observed that state-of-the-art ASR systems which deliver impressive benchmark
results, struggle with speakers of certain regions or demographics due to
variation in their speech properties. In this work, we describe the curation of
a massive speech dataset of 8740 hours consisting of $\sim9.8$K technical
lectures in the English language along with their transcripts delivered by
instructors representing various parts of Indian demography. The dataset is
sourced from the very popular NPTEL MOOC platform. We use the curated dataset
to measure the existing disparity in YouTube Automatic Captions and OpenAI
Whisper model performance across the diverse demographic traits of speakers in
India. While there exists disparity due to gender, native region, age and
speech rate of speakers, disparity based on caste is non-existent. We also
observe statistically significant disparity across the disciplines of the
lectures. These results indicate the need of more inclusive and robust ASR
systems and more representational datasets for disparity evaluation in them.