Using Scene and Semantic Features for Multi-modal Emotion Recognition
Zhifeng Wang Ramesh Sankaranarayana
Automatic emotion recognition is a hot topic with a wide range of
applications. Much work has been done in the area of automatic emotion
recognition in recent years. The focus has been mainly on using the
characteristics of a person such as speech, facial expression and pose for this
purpose. However, the processing of scene and semantic features for emotion
recognition has had limited exploration. In this paper, we propose to use
combined scene and semantic features, along with personal features, for
multi-modal emotion recognition. Scene features will describe the environment
or context in which the target person is operating. The semantic feature can
include objects that are present in the environment, as well as their
attributes and relationships with the target person. In addition, we use a
modified EmbraceNet to extract features from the images, which is trained to
learn both the body and pose features simultaneously. By fusing both body and
pose features, the EmbraceNet can improve the accuracy and robustness of the
model, particularly when dealing with partially missing data. This is because
having both body and pose features provides a more complete representation of
the subject in the images, which can help the model to make more accurate
predictions even when some parts of body are missing. We demonstrate the
efficiency of our method on the benchmark EMOTIC dataset. We report an average
precision of 40.39\% across the 26 emotion categories, which is a 5\%
improvement over previous approaches.