Speech Swin-Transformer: Exploring a Hierarchical Transformer with
  Shifted Windows for Speech Emotion Recognition
Yong Wang Cheng Lu Hailun Lian Yan Zhao Björn Schuller Yuan Zong Wenming Zheng
Swin-Transformer has demonstrated remarkable success in computer vision by
leveraging its hierarchical feature representation based on Transformer. In
speech signals, emotional information is distributed across different scales of
speech features, e.\,g., word, phrase, and utterance. Drawing above
inspiration, this paper presents a hierarchical speech Transformer with shifted
windows to aggregate multi-scale emotion features for speech emotion
recognition (SER), called Speech Swin-Transformer. Specifically, we first
divide the speech spectrogram into segment-level patches in the time domain,
composed of multiple frame patches. These segment-level patches are then
encoded using a stack of Swin blocks, in which a local window Transformer is
utilized to explore local inter-frame emotional information across frame
patches of each segment patch. After that, we also design a shifted window
Transformer to compensate for patch correlations near the boundaries of segment
patches. Finally, we employ a patch merging operation to aggregate
segment-level emotional features for hierarchical speech representation by
expanding the receptive field of Transformer from frame-level to segment-level.
Experimental results demonstrate that our proposed Speech Swin-Transformer
outperforms the state-of-the-art methods.