Layer-Wise Analysis of Self-Supervised Acoustic Word Embeddings: A Study
  on Speech Emotion Recognition
Alexandra Saliba Yuanchao Li Ramon Sanabria Catherine Lai
The efficacy of self-supervised speech models has been validated, yet the
optimal utilization of their representations remains challenging across diverse
tasks. In this study, we delve into Acoustic Word Embeddings (AWEs), a
fixed-length feature derived from continuous representations, to explore their
advantages in specific tasks. AWEs have previously shown utility in capturing
acoustic discriminability. In light of this, we propose measuring layer-wise
similarity between AWEs and word embeddings, aiming to further investigate the
inherent context within AWEs. Moreover, we evaluate the contribution of AWEs,
in comparison to other types of speech features, in the context of Speech
Emotion Recognition (SER). Through a comparative experiment and a layer-wise
accuracy analysis on two distinct corpora, IEMOCAP and ESD, we explore
differences between AWEs and raw self-supervised representations, as well as
the proper utilization of AWEs alone and in combination with word embeddings.
Our findings underscore the acoustic context conveyed by AWEs and showcase the
highly competitive SER accuracies by appropriately employing AWEs.