Utilizing Whisper to Enhance Multi-Branched Speech Intelligibility
  Prediction Model for Hearing Aids
Ryandhimas E. Zezario Fei Chen Chiou-Shann Fuh Hsin-Min Wang Yu Tsao
Automated assessment of speech intelligibility in hearing aid (HA) devices is
of great importance. Our previous work introduced a non-intrusive
multi-branched speech intelligibility prediction model called MBI-Net, which
achieved top performance in the Clarity Prediction Challenge 2022. Based on the
promising results of the MBI-Net model, we aim to further enhance its
performance by leveraging Whisper embeddings to enrich acoustic features. In
this study, we propose two improved models, namely MBI-Net+ and MBI-Net++.
MBI-Net+ maintains the same model architecture as MBI-Net, but replaces
self-supervised learning (SSL) speech embeddings with Whisper embeddings to
deploy cross-domain features. On the other hand, MBI-Net++ further employs a
more elaborate design, incorporating an auxiliary task to predict frame-level
and utterance-level scores of the objective speech intelligibility metric HASPI
(Hearing Aid Speech Perception Index) and multi-task learning. Experimental
results confirm that both MBI-Net++ and MBI-Net+ achieve better prediction
performance than MBI-Net in terms of multiple metrics, and MBI-Net++ is better
than MBI-Net+.