Facial Emotion Recognition Under Mask Coverage Using a Data Augmentation
  Technique
Aref Farhadipour Pouya Taghipour
Identifying human emotions using AI-based computer vision systems, when
individuals wear face masks, presents a new challenge in the current Covid-19
pandemic. In this study, we propose a facial emotion recognition system capable
of recognizing emotions from individuals wearing different face masks. A novel
data augmentation technique was utilized to improve the performance of our
model using four mask types for each face image. We evaluated the effectiveness
of four convolutional neural networks, Alexnet, Squeezenet, Resnet50 and
VGGFace2 that were trained using transfer learning. The experimental findings
revealed that our model works effectively in multi-mask mode compared to
single-mask mode. The VGGFace2 network achieved the highest accuracy rate, with
97.82% for the person-dependent mode and 74.21% for the person-independent mode
using the JAFFE dataset. However, we evaluated our proposed model using the
UIBVFED dataset. The Resnet50 has demonstrated superior performance, with
accuracies of 73.68% for the person-dependent mode and 59.57% for the
person-independent mode. Moreover, we employed metrics such as precision,
sensitivity, specificity, AUC, F1 score, and confusion matrix to measure our
system's efficiency in detail. Additionally, the LIME algorithm was used to
visualize CNN's decision-making strategy.