Multi Loss-based Feature Fusion and Top Two Voting Ensemble Decision
  Strategy for Facial Expression Recognition in the Wild
Guangyao Zhou Yuanlun Xie Wenhong Tian
Facial expression recognition (FER) in the wild is a challenging task
affected by the image quality and has attracted broad interest in computer
vision. There is no research using feature fusion and ensemble strategy for FER
simultaneously. Different from previous studies, this paper applies both
internal feature fusion for a single model and feature fusion among multiple
networks, as well as the ensemble strategy. This paper proposes one novel
single model named R18+FAML, as well as one ensemble model named
R18+FAML-FGA-T2V to improve the performance of the FER in the wild. Based on
the structure of ResNet18 (R18), R18+FAML combines internal Feature fusion and
three Attention blocks using Multiple Loss functions (FAML) to improve the
diversity of the feature extraction. To improve the performance of R18+FAML, we
propose a Feature fusion among networks based on the Genetic Algorithm (FGA),
which can fuse the convolution kernels for feature extraction of multiple
networks. On the basis of R18+FAML and FGA, we propose one ensemble strategy,
i.e., the Top Two Voting (T2V) to support the classification of FER, which can
consider more classification information comprehensively. Combining the above
strategies, R18+FAML-FGA-T2V can focus on the main expression-aware areas.
Extensive experiments demonstrate that our single model R18+FAML and the
ensemble model R18+FAML-FGA-T2V achieve the accuracies of $\left( 90.32, 62.17,
65.83 \right)\%$ and $\left( 91.59, 63.27, 66.63 \right)\%$ on three
challenging unbalanced FER datasets RAF-DB, AffectNet-8 and AffectNet-7
respectively, both outperforming the state-of-the-art results.