A$^{3}$lign-DFER: Pioneering Comprehensive Dynamic Affective Alignment
  for Dynamic Facial Expression Recognition with CLIP
Zeng Tao Yan Wang Junxiong Lin Haoran Wang Xinji Mai Jiawen Yu Xuan Tong Ziheng Zhou Shaoqi Yan Qing Zhao Liyuan Han Wenqiang Zhang
The performance of CLIP in dynamic facial expression recognition (DFER) task
doesn't yield exceptional results as observed in other CLIP-based
classification tasks. While CLIP's primary objective is to achieve alignment
between images and text in the feature space, DFER poses challenges due to the
abstract nature of text and the dynamic nature of video, making label
representation limited and perfect alignment difficult. To address this issue,
we have designed A$^{3}$lign-DFER, which introduces a new DFER labeling
paradigm to comprehensively achieve alignment, thus enhancing CLIP's
suitability for the DFER task. Specifically, our A$^{3}$lign-DFER method is
designed with multiple modules that work together to obtain the most suitable
expanded-dimensional embeddings for classification and to achieve alignment in
three key aspects: affective, dynamic, and bidirectional. We replace the input
label text with a learnable Multi-Dimensional Alignment Token (MAT), enabling
alignment of text to facial expression video samples in both affective and
dynamic dimensions. After CLIP feature extraction, we introduce the Joint
Dynamic Alignment Synchronizer (JAS), further facilitating synchronization and
alignment in the temporal dimension. Additionally, we implement a Bidirectional
Alignment Training Paradigm (BAP) to ensure gradual and steady training of
parameters for both modalities. Our insightful and concise A$^{3}$lign-DFER
method achieves state-of-the-art results on multiple DFER datasets, including
DFEW, FERV39k, and MAFW. Extensive ablation experiments and visualization
studies demonstrate the effectiveness of A$^{3}$lign-DFER. The code will be
available in the future.