MSSTNet: A Multi-Scale Spatio-Temporal CNN-Transformer Network for
  Dynamic Facial Expression Recognition
Linhuang Wang Xin Kang Fei Ding Satoshi Nakagawa Fuji Ren
Unlike typical video action recognition, Dynamic Facial Expression
Recognition (DFER) does not involve distinct moving targets but relies on
localized changes in facial muscles. Addressing this distinctive attribute, we
propose a Multi-Scale Spatio-temporal CNN-Transformer network (MSSTNet). Our
approach takes spatial features of different scales extracted by CNN and feeds
them into a Multi-scale Embedding Layer (MELayer). The MELayer extracts
multi-scale spatial information and encodes these features before sending them
into a Temporal Transformer (T-Former). The T-Former simultaneously extracts
temporal information while continually integrating multi-scale spatial
information. This process culminates in the generation of multi-scale
spatio-temporal features that are utilized for the final classification. Our
method achieves state-of-the-art results on two in-the-wild datasets.
Furthermore, a series of ablation experiments and visualizations provide
further validation of our approach's proficiency in leveraging spatio-temporal
information within DFER.