HTNet for micro-expression recognition
Zhifeng Wang Kaihao Zhang Wenhan Luo Ramesh Sankaranarayana
Facial expression is related to facial muscle contractions and different
muscle movements correspond to different emotional states. For micro-expression
recognition, the muscle movements are usually subtle, which has a negative
impact on the performance of current facial emotion recognition algorithms.
Most existing methods use self-attention mechanisms to capture relationships
between tokens in a sequence, but they do not take into account the inherent
spatial relationships between facial landmarks. This can result in sub-optimal
performance on micro-expression recognition tasks.Therefore, learning to
recognize facial muscle movements is a key challenge in the area of
micro-expression recognition. In this paper, we propose a Hierarchical
Transformer Network (HTNet) to identify critical areas of facial muscle
movement. HTNet includes two major components: a transformer layer that
leverages the local temporal features and an aggregation layer that extracts
local and global semantical facial features. Specifically, HTNet divides the
face into four different facial areas: left lip area, left eye area, right eye
area and right lip area. The transformer layer is used to focus on representing
local minor muscle movement with local self-attention in each area. The
aggregation layer is used to learn the interactions between eye areas and lip
areas. The experiments on four publicly available micro-expression datasets
show that the proposed approach outperforms previous methods by a large margin.
The codes and models are available at:
\url{https://github.com/wangzhifengharrison/HTNet}