AUD-TGN: Advancing Action Unit Detection with Temporal Convolution and
  GPT-2 in Wild Audiovisual Contexts
Jun Yu Zerui Zhang Zhihong Wei Gongpeng Zhao Zhongpeng Cai Yongqi Wang Guochen Xie Jichao Zhu Wangyuan Zhu
Leveraging the synergy of both audio data and visual data is essential for
understanding human emotions and behaviors, especially in in-the-wild setting.
Traditional methods for integrating such multimodal information often stumble,
leading to less-than-ideal outcomes in the task of facial action unit
detection. To overcome these shortcomings, we propose a novel approach
utilizing audio-visual multimodal data. This method enhances audio feature
extraction by leveraging Mel Frequency Cepstral Coefficients (MFCC) and Log-Mel
spectrogram features alongside a pre-trained VGGish network. Moreover, this
paper adaptively captures fusion features across modalities by modeling the
temporal relationships, and ultilizes a pre-trained GPT-2 model for
sophisticated context-aware fusion of multimodal information. Our method
notably improves the accuracy of AU detection by understanding the temporal and
contextual nuances of the data, showcasing significant advancements in the
comprehension of intricate scenarios. These findings underscore the potential
of integrating temporal dynamics and contextual interpretation, paving the way
for future research endeavors.