Transfer Learning from Whisper for Microscopic Intelligibility
  Prediction
Paul Best Santiago Cuervo Ricard Marxer
Macroscopic intelligibility models predict the expected human word-error-rate
for a given speech-in-noise stimulus. In contrast, microscopic intelligibility
models aim to make fine-grained predictions about listeners' perception, e.g.
predicting phonetic or lexical responses. State-of-the-art macroscopic models
use transfer learning from large scale deep learning models for speech
processing, whereas such methods have rarely been used for microscopic
modeling. In this paper, we study the use of transfer learning from Whisper, a
state-of-the-art deep learning model for automatic speech recognition, for
microscopic intelligibility prediction at the level of lexical responses. Our
method outperforms the considered baselines, even in a zero-shot setup, and
yields a relative improvement of up to 66\% when fine-tuned to predict
listeners' responses. Our results showcase the promise of large scale deep
learning based methods for microscopic intelligibility prediction.