A Review on Automatic Facial Expression Recognition Systems Assisted by Multimodal Sensor Data
---
paper_title: Survey on RGB, 3D, Thermal, and Multimodal Approaches for Facial Expression Recognition: History, Trends, and Affect-Related Applications

paper_content:

Facial expressions are an important way through which humans interact socially. Building a system capable of automatically recognizing facial expressions from images and video has been an intense field of study in recent years. Interpreting such expressions remains challenging and much research is needed about the way they relate to human affect. This paper presents a general overview of automatic RGB, 3D, thermal and multimodal facial expression analysis. We define a new taxonomy for the field, encompassing all steps from face detection to facial expression recognition, and describe and classify the state of the art methods accordingly. We also present the important datasets and the bench-marking of most influential methods. We conclude with a general discussion about trends, important questions and future lines of research.
---
paper_title: Emotion In The Human Face Guidelines For Research And An Integration Of Findings

paper_content:

Thank you for reading emotion in the human face guidelines for research and an integration of findings. As you may know, people have look numerous times for their favorite readings like this emotion in the human face guidelines for research and an integration of findings, but end up in harmful downloads. Rather than reading a good book with a cup of coffee in the afternoon, instead they cope with some harmful virus inside their computer.
---
paper_title: Augmented reality-based self-facial modeling to promote the emotional expression and social skills of adolescents with autism spectrum disorders.

paper_content:

Autism spectrum disorders (ASD) are characterized by a reduced ability to understand the emotions of other people; this ability involves recognizing facial expressions. This study assessed the possibility of enabling three adolescents with ASD to become aware of facial expressions observed in situations in a school setting simulated using augmented reality (AR) technology. The AR system provided three-dimensional (3-D) animations of six basic facial expressions overlaid on participant faces to facilitate practicing emotional judgments and social skills. Based on the multiple baseline design across subjects, the data indicated that AR intervention can improve the appropriate recognition and response to facial emotional expressions seen in the situational task.
---
paper_title: Posed and spontaneous facial expression differentiation using deep Boltzmann machines

paper_content:

Current works on differentiating between posed and spontaneous facial expressions usually use features that are handcrafted for expression category recognition. Till now, no features have been specifically designed for differentiating between posed and spontaneous facial expressions. Recently, deep learning models have been proven to be efficient for many challenging computer vision tasks, and therefore in this paper we propose using the deep Boltzmann machine to learn representations of facial images and to differentiate between posed and spontaneous facial expressions. First, faces are located from images. Then, a two-layer deep Boltzmann machine is trained to distinguish posed and spon-tanous expressions. Experimental results on two benchmark datasets, i.e. the SPOS and USTC-NVIE datasets, demonstrate that the deep Boltzmann machine performs well on posed and spontaneous expression differentiation tasks. Comparison results on both datasets show that our method has an advantage over the other methods.
---
paper_title: Nonverbal leakage and clues to deception.

paper_content:

Abstract : Research relevant to psychotherapy regarding facial expression and body movement, has shown that the kind of information which can be gleaned from the patients words - information about affects, attitudes, interpersonal styles, psychodynamics - can also be derived from his concomitant nonverbal behavior. The study explores the interaction situation, and considers how within deception interactions differences in neuroanatomy and cultural influences combine to produce specific types of body movements and facial expressions which escape efforts to deceive and emerge as leakage or deception clues.
---
paper_title: Learning Social Relation Traits from Face Images

paper_content:

Social relation defines the association, e.g, warm, friendliness, and dominance, between two or more people. Motivated by psychological studies, we investigate if such fine-grained and high-level relation traits can be characterised and quantified from face images in the wild. To address this challenging problem we propose a deep model that learns a rich face representation to capture gender, expression, head pose, and age-related attributes, and then performs pairwise-face reasoning for relation prediction. To learn from heterogeneous attribute sources, we formulate a new network architecture with a bridging layer to leverage the inherent correspondences among these datasets. It can also cope with missing target attribute labels. Extensive experiments show that our approach is effective for fine-grained social relation learning in images and videos.
---
paper_title: A Survey on Human Face Expression Recognition Techniques

paper_content:

Abstract Human Face expression Recognition is one of the most powerful and challenging tasks in social communication. Generally, face expressions are natural and direct means for human beings to communicate their emotions and intentions. Face expressions are the key characteristics of non-verbal communication. This paper describes the survey of Face Expression Recognition (FER) techniques which include the three major stages such as preprocessing, feature extraction and classification. This survey explains the various types of FER techniques with its major contributions. The performance of various FER techniques is compared based on the number of expressions recognized and complexity of algorithms. Databases like JAFFE, CK, and some other variety of facial expression databases are discussed in this survey. The study on classifiers gather from recent papers reveals a more powerful and reliable understanding of the peculiar characteristics of classifiers for research fellows.
---
paper_title: A systematic survey of facial expression recognition techniques

paper_content:

Facial expression recognition (FER) is an imperative part in the exploration of man-machine interface. Presently, it is applied in a wide variety of environments including robotics, mobile application, digital signs etc. Since faces can divulge the emotional sentiment of a person, automatic elucidation of facial expressions delivers a natural way to interact with computers. FER systems scrutinize facial images captured via computer vision to detect human expressions or interpret emotion thereby. In this paper, we first explain the process of FER which will further assist how a particular technique is modelled. Also an extensive survey of the latest techniques for FER has been presented comparing them based on their feature extraction technique, classification method and recognition rate.
---
paper_title: Facial Expression Recognition in Adolescents With Mood and Anxiety Disorders

paper_content:

OBJECTIVE: The authors examined facial expression recognition in adolescents with mood and anxiety disorders. METHOD: Standard facial emotion identification tests were given to youth with bipolar disorder (N=11) or DSM-IV anxiety disorders (N=10) and a group of healthy comparison subjects (N=25). RESULTS: Relative to the anxiety disorder and healthy comparison groups, the subjects with bipolar disorder made more emotion recognition errors when presented with faces of children. Unlike the anxious and comparison subjects, bipolar disorder youth were prone to misidentify faces as angry. No differences in emotion recognition errors were seen when the adolescents were presented with adult faces. CONCLUSIONS: A bias to misinterpret the facial expressions of peers as angry may characterize youth with bipolar disorder but not youth with anxiety disorders. This bias may relate to social impairment in youth with bipolar disorder.
---
paper_title: Automatic Analysis of Facial Affect: A Survey of Registration, Representation, and Recognition

paper_content:

Automatic affect analysis has attracted great interest in various contexts including the recognition of action units and basic or non-basic emotions. In spite of major efforts, there are several open questions on what the important cues to interpret facial expressions are and how to encode them. In this paper, we review the progress across a range of affect recognition applications to shed light on these fundamental questions. We analyse the state-of-the-art solutions by decomposing their pipelines into fundamental components, namely face registration, representation, dimensionality reduction and recognition. We discuss the role of these components and highlight the models and new trends that are followed in their design. Moreover, we provide a comprehensive analysis of facial representations by uncovering their advantages and limitations; we elaborate on the type of information they encode and discuss how they deal with the key challenges of illumination variations, registration errors, head-pose variations, occlusions, and identity bias. This survey allows us to identify open issues and to define future directions for designing real-world affect recognition systems.
---
paper_title: An investigation of basic facial expression recognition in autism spectrum disorders

paper_content:

This study was designed to test three competing hypotheses (impaired configural processing; impaired Theory of Mind; atypical amygdala functioning) to explain the basic facial expression recognition profile of adults with autism spectrum disorders (ASD). In Experiment 1 the Ekman and Friesen (1976) series were presented upright and inverted. Individuals with ASD were significantly less accurate than controls at recognising upright facial expressions of fear, sadness and disgust and their pattern of errors suggested some configural processing difficulties. Impaired recognition of inverted facial expressions suggested some additional difficulties processing the facial features. Unexpectedly, the clinical group misidentified fear as anger. In Experiment 2 feature processing of facial expressions was investigated by presenting stimuli in a piecemeal fashion, starting with either just the eyes or the mouth. Individuals with ASD were impaired at recognising fear from the eyes and disgust from the mouth; they al...
---
paper_title: Facial Emotion Recognition: A Survey and Real-World User Experiences in Mixed Reality

paper_content:

Extensive possibilities of applications have made emotion recognition ineluctable and challenging in the field of computer science. The use of non-verbal cues such as gestures, body movement, and facial expressions convey the feeling and the feedback to the user. This discipline of Human–Computer Interaction places reliance on the algorithmic robustness and the sensitivity of the sensor to ameliorate the recognition. Sensors play a significant role in accurate detection by providing a very high-quality input, hence increasing the efficiency and the reliability of the system. Automatic recognition of human emotions would help in teaching social intelligence in the machines. This paper presents a brief study of the various approaches and the techniques of emotion recognition. The survey covers a succinct review of the databases that are considered as data sets for algorithms detecting the emotions by facial expressions. Later, mixed reality device Microsoft HoloLens (MHL) is introduced for observing emotion recognition in Augmented Reality (AR). A brief introduction of its sensors, their application in emotion recognition and some preliminary results of emotion recognition using MHL are presented. The paper then concludes by comparing results of emotion recognition by the MHL and a regular webcam.
---
paper_title: Static and dynamic 3D facial expression recognition: A comprehensive survey☆

paper_content:

Automatic facial expression recognition constitutes an active research field due to the latest advances in computing technology that make the user's experience a clear priority. The majority of work conducted in this area involves 2D imagery, despite the problems this presents due to inherent pose and illumination variations. In order to deal with these problems, 3D and 4D (dynamic 3D) recordings are increasingly used in expression analysis research. In this paper we survey the recent advances in 3D and 4D facial expression recognition. We discuss developments in 3D facial data acquisition and tracking, and present currently available 3D/4D face databases suitable for 3D/4D facial expressions analysis as well as the existing facial expression recognition systems that exploit either 3D or 4D data in detail. Finally, challenges that have to be addressed if 3D facial expression recognition systems are to become a part of future applications are extensively discussed.
---
paper_title: Facial expression recognition in still pictures and videos using active appearance models: a comparison approach

paper_content:

The paper highlights the performance of video sequence-oriented facial expression recognition using Active Appearance Model -- AAM, in a comparison with the analysis based on still pictures. The AAM is used to extract relevant information regarding the shapes of the faces to be analyzed. Specific key points from a Facial Characteristic Point - FCP model are used to derive the set of features. These features are used for the classification of the expressions of a new face sample into the prototypic emotions. The classification method uses Support Vector Machines.
---
paper_title: Human facial expressions as adaptations: Evolutionary questions in facial expression research

paper_content:

The importance of the face in social interaction and social intelligence is widely recognized in anthropology. Yet the adaptive functions of human facial expression remain largely unknown. An evolutionary model of human facial expression as behavioral adaptation can be constructed, given the current knowledge of the phenotypic variation, ecological contexts, and fitness consequences of facial behavior. Studies of facial expression are available, but results are not typically framed in an evolutionary perspective. This review identifies the relevant physical phenomena of facial expression and integrates the study of this behavior with the anthropological study of communication and sociality in general. Anthropological issues with relevance to the evolutionary study of facial expression include: facial expressions as coordinated, stereotyped behavioral phenotypes, the unique contexts and functions of different facial expressions, the relationship of facial expression to speech, the value of facial expressions as signals, and the relationship of facial expression to social intelligence in humans and in nonhuman primates. Human smiling is used as an example of adaptation, and testable hypotheses concerning the human smile, as well as other expressions, are proposed.
---
paper_title: Eyemotion: Classifying Facial Expressions in VR Using Eye-Tracking Cameras

paper_content:

One of the main challenges of social interaction in virtual reality settings is that head-mounted displays occlude a large portion of the face, blocking facial expressions and thereby restricting social engagement cues among users. Hence, auxiliary means of sensing and conveying these expressions are needed. We present an algorithm to automatically infer expressions by analyzing only a partially occluded face while the user is engaged in a virtual reality experience. Specifically, we show that images of the user's eyes captured from an IR gaze-tracking camera within a VR headset are sufficient to infer a select subset of facial expressions without the use of any fixed external camera. Using these inferences, we can generate dynamic avatars in real-time which function as an expressive surrogate for the user. We propose a novel data collection pipeline as well as a novel approach for increasing CNN accuracy via personalization. Our results show a mean accuracy of 74% ($F1$ of 0.73) among 5 `emotive' expressions and a mean accuracy of 70% ($F1$ of 0.68) among 10 distinct facial action units, outperforming human raters.
---
paper_title: Study of video based facial expression and emotions recognition methods

paper_content:

In real life scenario, facial expressions and emotions are nothing but responses to the external and internal events of human being. In human computer interaction, recognition of end user's expressions and emotions from the video streaming plays very important role. In such systems it is required to track the dynamic changes in human face movements quickly in order to deliver the required response system. The one real time application is physical fatigue detection based on facial detection and expressions such as driver fatigue detection in order to prevent the accidents on road. Face expression based physical fatigue analysis or detection is out of scope of this paper, but this paper reveal study on different methods those are presented recently for facial expression and/or emotions recognition using video. This paper presenting the methodologies in terms of feature extraction and classification used in facial expression and/or emotion recognition methods with their comparative study. The comparative study is done based on accuracy, implementation tool, advantages and disadvantages. The outcome of this paper is the current research gap and research challenges those are still open to solve for video based facial detection and recognition systems. The survey on recent methods is appropriately presented throughout this paper by considering future research works.
---
paper_title: Survey on real-time facial expression recognition techniques

paper_content:

Cameras constantly capture and track facial images and videos on cell phones, webcams etc. In the past decade, facial expression classification and recognition has been the topic of interest as facial expression analysis has a wide range of applications such as intelligent tutoring system, systems for psychological studies etc. This study reviews the latest advances in the algorithms and techniques used in distinct phases of real-time facial expression recognition. Though there are state-of-art approaches to address facial expression identification in real-time, many issues such as subjectivity-removal, occlusion, pose, low resolution, scale, variations in illumination level and identification of baseline frame still remain unaddressed. Attempts to deal with such issues for higher accuracy lead to a trade-off in efficiency. Furthermore, the goal of this study is to elaborate on these issues and highlight the solutions provided by the current approaches. This survey has helped the authors to understand that there is a need for a better strategy to address these issues without having to trade-off performance in real-time.
---
paper_title: Cognitive architectures for human-robot interaction

paper_content:

Developments in autonomous agents for Human-Robot Interaction (HRI), particularly social, are gathering pace. The typical approach to such efforts is to start with an application to a specific interaction context (problem, task, or aspect of interaction) and then try to generalise to different contexts. Alternatively however, the application of Cognitive Architectures emphasises generality across contexts in the first instance. While not the“silver-bullet”solution, this perspective has a number of advantages both in terms of the functionality of the resulting systems, and indeed in the process of applying these ideas. Centred on invited talks to present a range of perspectives, this workshop provides a forum to introduce and discuss the application (both existing and potential) of Cognitive Architectures to HRI, particularly in the social domain. Participants will gain insight into how such a consideration of Cognitive Architectures complements the development of autonomous social robots. Categories and Subject Descriptors C.0 [Computer Systems Organization]: General; I.2.0 [Artificial Intelligence]: General—cognitive simulation General Terms Design, Human Factors, Theory
---
paper_title: Deeply Learning Deformable Facial Action Parts Model for Dynamic Expression Analysis

paper_content:

Expressions are facial activities invoked by sets of muscle motions, which would give rise to large variations in appearance mainly around facial parts. Therefore, for visual-based expression analysis, localizing the action parts and encoding them effectively become two essential but challenging problems. To take them into account jointly for expression analysis, in this paper, we propose to adapt 3D Convolutional Neural Networks (3D CNN) with deformable action parts constraints. Specifically, we incorporate a deformable parts learning component into the 3D CNN framework, which can detect specific facial action parts under the structured spatial constraints, and obtain the discriminative part-based representation simultaneously. The proposed method is evaluated on two posed expression datasets, CK+, MMI, and a spontaneous dataset FERA. We show that, besides achieving state-of-the-art expression recognition accuracy, our method also enjoys the intuitive appeal that the part detection map can desirably encode the mid-level semantics of different facial action parts.
---
paper_title: Survey on RGB, 3D, Thermal, and Multimodal Approaches for Facial Expression Recognition: History, Trends, and Affect-Related Applications

paper_content:

Facial expressions are an important way through which humans interact socially. Building a system capable of automatically recognizing facial expressions from images and video has been an intense field of study in recent years. Interpreting such expressions remains challenging and much research is needed about the way they relate to human affect. This paper presents a general overview of automatic RGB, 3D, thermal and multimodal facial expression analysis. We define a new taxonomy for the field, encompassing all steps from face detection to facial expression recognition, and describe and classify the state of the art methods accordingly. We also present the important datasets and the bench-marking of most influential methods. We conclude with a general discussion about trends, important questions and future lines of research.
---
paper_title: A Survey on Human Face Expression Recognition Techniques

paper_content:

Abstract Human Face expression Recognition is one of the most powerful and challenging tasks in social communication. Generally, face expressions are natural and direct means for human beings to communicate their emotions and intentions. Face expressions are the key characteristics of non-verbal communication. This paper describes the survey of Face Expression Recognition (FER) techniques which include the three major stages such as preprocessing, feature extraction and classification. This survey explains the various types of FER techniques with its major contributions. The performance of various FER techniques is compared based on the number of expressions recognized and complexity of algorithms. Databases like JAFFE, CK, and some other variety of facial expression databases are discussed in this survey. The study on classifiers gather from recent papers reveals a more powerful and reliable understanding of the peculiar characteristics of classifiers for research fellows.
---
paper_title: A systematic survey of facial expression recognition techniques

paper_content:

Facial expression recognition (FER) is an imperative part in the exploration of man-machine interface. Presently, it is applied in a wide variety of environments including robotics, mobile application, digital signs etc. Since faces can divulge the emotional sentiment of a person, automatic elucidation of facial expressions delivers a natural way to interact with computers. FER systems scrutinize facial images captured via computer vision to detect human expressions or interpret emotion thereby. In this paper, we first explain the process of FER which will further assist how a particular technique is modelled. Also an extensive survey of the latest techniques for FER has been presented comparing them based on their feature extraction technique, classification method and recognition rate.
---
paper_title: Automatic Analysis of Facial Affect: A Survey of Registration, Representation, and Recognition

paper_content:

Automatic affect analysis has attracted great interest in various contexts including the recognition of action units and basic or non-basic emotions. In spite of major efforts, there are several open questions on what the important cues to interpret facial expressions are and how to encode them. In this paper, we review the progress across a range of affect recognition applications to shed light on these fundamental questions. We analyse the state-of-the-art solutions by decomposing their pipelines into fundamental components, namely face registration, representation, dimensionality reduction and recognition. We discuss the role of these components and highlight the models and new trends that are followed in their design. Moreover, we provide a comprehensive analysis of facial representations by uncovering their advantages and limitations; we elaborate on the type of information they encode and discuss how they deal with the key challenges of illumination variations, registration errors, head-pose variations, occlusions, and identity bias. This survey allows us to identify open issues and to define future directions for designing real-world affect recognition systems.
---
paper_title: Facial Emotion Recognition: A Survey and Real-World User Experiences in Mixed Reality

paper_content:

Extensive possibilities of applications have made emotion recognition ineluctable and challenging in the field of computer science. The use of non-verbal cues such as gestures, body movement, and facial expressions convey the feeling and the feedback to the user. This discipline of Human–Computer Interaction places reliance on the algorithmic robustness and the sensitivity of the sensor to ameliorate the recognition. Sensors play a significant role in accurate detection by providing a very high-quality input, hence increasing the efficiency and the reliability of the system. Automatic recognition of human emotions would help in teaching social intelligence in the machines. This paper presents a brief study of the various approaches and the techniques of emotion recognition. The survey covers a succinct review of the databases that are considered as data sets for algorithms detecting the emotions by facial expressions. Later, mixed reality device Microsoft HoloLens (MHL) is introduced for observing emotion recognition in Augmented Reality (AR). A brief introduction of its sensors, their application in emotion recognition and some preliminary results of emotion recognition using MHL are presented. The paper then concludes by comparing results of emotion recognition by the MHL and a regular webcam.
---
paper_title: Static and dynamic 3D facial expression recognition: A comprehensive survey☆

paper_content:

Automatic facial expression recognition constitutes an active research field due to the latest advances in computing technology that make the user's experience a clear priority. The majority of work conducted in this area involves 2D imagery, despite the problems this presents due to inherent pose and illumination variations. In order to deal with these problems, 3D and 4D (dynamic 3D) recordings are increasingly used in expression analysis research. In this paper we survey the recent advances in 3D and 4D facial expression recognition. We discuss developments in 3D facial data acquisition and tracking, and present currently available 3D/4D face databases suitable for 3D/4D facial expressions analysis as well as the existing facial expression recognition systems that exploit either 3D or 4D data in detail. Finally, challenges that have to be addressed if 3D facial expression recognition systems are to become a part of future applications are extensively discussed.
---
paper_title: Study of video based facial expression and emotions recognition methods

paper_content:

In real life scenario, facial expressions and emotions are nothing but responses to the external and internal events of human being. In human computer interaction, recognition of end user's expressions and emotions from the video streaming plays very important role. In such systems it is required to track the dynamic changes in human face movements quickly in order to deliver the required response system. The one real time application is physical fatigue detection based on facial detection and expressions such as driver fatigue detection in order to prevent the accidents on road. Face expression based physical fatigue analysis or detection is out of scope of this paper, but this paper reveal study on different methods those are presented recently for facial expression and/or emotions recognition using video. This paper presenting the methodologies in terms of feature extraction and classification used in facial expression and/or emotion recognition methods with their comparative study. The comparative study is done based on accuracy, implementation tool, advantages and disadvantages. The outcome of this paper is the current research gap and research challenges those are still open to solve for video based facial detection and recognition systems. The survey on recent methods is appropriately presented throughout this paper by considering future research works.
---
paper_title: Survey on real-time facial expression recognition techniques

paper_content:

Cameras constantly capture and track facial images and videos on cell phones, webcams etc. In the past decade, facial expression classification and recognition has been the topic of interest as facial expression analysis has a wide range of applications such as intelligent tutoring system, systems for psychological studies etc. This study reviews the latest advances in the algorithms and techniques used in distinct phases of real-time facial expression recognition. Though there are state-of-art approaches to address facial expression identification in real-time, many issues such as subjectivity-removal, occlusion, pose, low resolution, scale, variations in illumination level and identification of baseline frame still remain unaddressed. Attempts to deal with such issues for higher accuracy lead to a trade-off in efficiency. Furthermore, the goal of this study is to elaborate on these issues and highlight the solutions provided by the current approaches. This survey has helped the authors to understand that there is a need for a better strategy to address these issues without having to trade-off performance in real-time.
---
paper_title: Transfer subspace learning for cross-dataset facial expression recognition

paper_content:

In this paper, we propose a transfer subspace learning approach cross-dataset facial expression recognition. To our best knowledge, this problem has been seldom addressed in the literature. While many facial expression recognition methods have been proposed in recent years, most of them assume that face images in the training and testing sets are collected under the same conditions so that they are independently and identically distributed. In many real applications, this assumption does not hold as the testing data are usually collected online and are generally more uncontrollable than the training data. Hence, the testing samples are likely different from the training samples. In this paper, we define this problem as cross-dataset facial expression recognition as the training and testing data are considered to be collected from different datasets due to different acquisition conditions. To address this, we propose a transfer subspace learning approach to learn a feature subspace which transfers the knowledge gained from the source domain (training samples) to the target domain (testing samples) to improve the recognition performance. To better exploit more complementary information for multiple feature representations of face images, we develop a multi-view transfer subspace learning approach where multiple different yet related subspaces are learned to transfer information from the source domain to the target domain. Experimental results are presented to demonstrate the efficacy of these proposed methods for the cross-dataset facial expression recognition task.
---
paper_title: GCV-Based Regularized Extreme Learning Machine for Facial Expression Recognition

paper_content:

Extreme learning machine (ELM) with a single-layer feed-forward network (SLFN) has acquired overwhelming attention. The structure of ELM has to be optimized through the incorporation of regularization to gain convenient results, and the Tikhonov regularization is frequently used. Regularization benefits in improving the generalized performance than traditional ELM. The estimation of regularization parameter mainly follows heuristic approaches or some empirical analysis through prior experience. When such a choice is not possible, the generalized cross-validation (GCV) method is one of the most popular choices for obtaining optimal regularization parameter. In this work, a new method of facial expression recognition is introduced where histogram of oriented gradients (HOG) feature extraction and GCV-based regularized ELM are applied. Experimental results on facial expression database JAFFE demonstrate promising performance which outperforms the other two classifiers, namely support vector machine (SVM) and k-nearest neighbor (KNN).
---
paper_title: A survey of affect recognition methods: audio, visual and spontaneous expressions

paper_content:

Automated analysis of human affective behavior has attracted increasing attention from researchers in psychology, computer science, linguistics, neuroscience, and related disciplines. Promising approaches have been reported, including automatic methods for facial and vocal affect recognition. However, the existing methods typically handle only deliberately displayed and exaggerated expressions of prototypical emotions-despite the fact that deliberate behavior differs in visual and audio expressions from spontaneously occurring behavior. Recently efforts to develop algorithms that can process naturally occurring human affective behavior have emerged. This paper surveys these efforts. We first discuss human emotion perception from a psychological perspective. Next, we examine the available approaches to solving the problem of machine understanding of human affective behavior occurring in real-world settings. We finally outline some scientific and engineering challenges for advancing human affect sensing technology.
---
paper_title: A Survey on Perception Methods for Human–Robot Interaction in Social Robots

paper_content:

For human–robot interaction (HRI), perception is one of the most important capabilities. This paper reviews several widely used perception methods of HRI in social robots. Specifically, we investigate general perception tasks crucial for HRI, such as where the objects are located in the rooms, what objects are in the scene, and how they interact with humans. We first enumerate representative social robots and summarize the most three important perception methods from these robots: feature extraction, dimensionality reduction, and semantic understanding. For feature extraction, four widely used signals including visual-based, audio-based, tactile-based and rang sensors-based are reviewed, and they are compared based on their advantages and disadvantages. For dimensionality reduction, representative methods including principle component analysis (PCA), linear discriminant analysis (LDA), and locality preserving projections (LPP) are reviewed. For semantic understanding, conventional techniques for several typical applications such as object recognition, object tracking, object segmentation, and speaker localization are discussed, and their characteristics and limitations are also analyzed. Moreover, several popular data sets used in social robotics and published semantic understanding results are analyzed and compared in light of our analysis of HRI perception methods. Lastly, we suggest important future work to analyze fundamental questions on perception methods in HRI.
---
paper_title: A survey on facial recognition based on local directional and local binary patterns

paper_content:

The goal of this research project was to come up with a combined face recognition algorithm which out-performs existing algorithms on accuracy. The identification of individuals using face recognition techniques is a challenging task. This is due to the variations resulting from facial expressions, makeup, rotations, illuminations and gestures. Facial images also contain a great deal of redundant information, which negatively affects the performance of the recognition system. This paper proposes a novel approach for recognizing facial images from facial features using feature descriptors, namely local binary patterns(LBP) and local directional patterns(LDP). This research work consisted of three parts, namely face representation, feature extraction and classification. The useful and unique features of the facial images were extracted in the feature extraction phase. In classification, the face image was compared with the images from the database. The face area was divided into small regions from which local binary and directional patterns (LBP/LDP) histograms were extracted and concatenated into a single feature vector(histogram). Experiments performed on the Cohn-Kanade facial expression database obtained a good recognition rate 99% indicating superiority of the proposed method compared to other methods. The proposed included a combination of local binary pattern(LBP) and local directional patterns(LDP+LBP+Voting Classifier) as the feature extractor and voting classifier classification algorithm which is an aggregate classifier composed of k-Nearest Neighbor, decision trees and support vector machines. The results showed improved accuracy results as compared to other local binary pattern variants in both scenarios where small datasets or huge datasets were used.
---
paper_title: Recognizing Action Units for Facial Expression Analysis

paper_content:

Most automatic expression analysis systems attempt to recognize a small set of prototypic expressions, such as happiness, anger, surprise, and fear. Such prototypic expressions, however, occur rather infrequently. Human emotions and intentions are more often communicated by changes in one or a few discrete facial features. In this paper, we develop an automatic face analysis (AFA) system to analyze facial expressions based on both permanent facial features (brows, eyes, mouth) and transient facial features (deepening of facial furrows) in a nearly frontal-view face image sequence. The AFA system recognizes fine-grained changes in facial expression into action units (AU) of the Facial Action Coding System (FACS), instead of a few prototypic expressions. Multistate face and facial component models are proposed for tracking and modeling the various facial features, including lips, eyes, brows, cheeks, and furrows. During tracking, detailed parametric descriptions of the facial features are extracted. With these parameters as the inputs, a group of action units (neutral expression, six upper face AU and 10 lower face AU) are recognized whether they occur alone or in combinations. The system has achieved average recognition rates of 96.4 percent (95.4 percent if neutral expressions are excluded) for upper face AU and 96.7 percent (95.6 percent with neutral expressions excluded) for lower face AU. The generalizability of the system has been tested by using independent image databases collected and FACS-coded for ground-truth by different research teams.
---
paper_title: Facial expression recognition via learning deep sparse autoencoders

paper_content:

Abstract Facial expression recognition is an important research issue in the pattern recognition field. In this paper, we intend to present a novel framework for facial expression recognition to automatically distinguish the expressions with high accuracy. Especially, a high-dimensional feature composed by the combination of the facial geometric and appearance features is introduced to the facial expression recognition due to its containing the accurate and comprehensive information of emotions. Furthermore, the deep sparse autoencoders (DSAE) are established to recognize the facial expressions with high accuracy by learning robust and discriminative features from the data. The experiment results indicate that the presented framework can achieve a high recognition accuracy of 95.79% on the extended Cohn–Kanade (CK+) database for seven facial expressions, which outperforms the other three state-of-the-art methods by as much as 3.17%, 4.09% and 7.41%, respectively. In particular, the presented approach is also applied to recognize eight facial expressions (including the neutral) and it provides a satisfactory recognition accuracy, which successfully demonstrates the feasibility and effectiveness of the approach in this paper.
---
paper_title: Facial expression recognition based on eigenspaces and principle component analysis

paper_content:

Facial expression detection or emotion recognition is one of the rising fields of research on intelligent systems. Emotion plays a significant role in non-verbal communication. An efficient face and facial feature detection algorithms plays as important role in indentifying of an emotion of a person at a particular moment. In this work, the authors implemented a system that recognises a person's facial expressions from the input images, using the algorithm of eigenspaces and principle component analysis (PCA). Eigenspaces are the face images which are projected onto a feature space that encodes the variation among known face images. PCA is used in this paper to make dimensional reduction of images in order to obtain a reduced representation of face images. The implementation is applied on three different facial expressions databases, extended Cohn-Kanade facial expression database, Japanese female facial expression database and self-made database in order to find out the effectiveness of the proposed method.
---
paper_title: Reliable facial expression recognition for multi-scale images using weber local binary image based cosine transform features

paper_content:

Accurate recognition of facial expression is a challenging problem especially from multi-scale and multi orientation face images. In this article, we propose a novel technique called Weber Local Binary Image Cosine Transform (WLBI-CT). WLBI-CT extracts and integrates the frequency components of images obtained through Weber local descriptor and local binary descriptor. These frequency components help in accurate classification of various facial expressions in the challenging domain of multi-scale and multi-orientation facial images. Identification of significant feature set plays a vital role in the success of any facial expression recognition system. Effect of multiple feature sets with varying block sizes has been investigated using different multi-scale images taken from well-known JAFEE, MMI and CK+ datasets. Extensive experimentation has been performed to demonstrate that the proposed technique outperforms the contemporary techniques in terms of recognition rate and computational time.
---
paper_title: Illumination Invariant Facial Expression Recognition Using Selected Merged Binary Patterns for Real World Images

paper_content:

Abstract Automatic facial expression recognition has always been a challenging task to understand human behavior from real world images. Certain type of issues are associated with such images that include poor illumination, different orientations and varying pose. The proposed technique first applies Fast Fourier Transform and Contrast Limited Adaptive Histogram Equalization (FFT + CLAHE) method to compensate the poor illumination. Then merged binary pattern code (MBPC) is generated for every pixel. Two bits per neighbourhood are produced to form a 16-bit code per pixel. This code merges local features to improve the effectiveness of facial expression recognition system. MBPC descriptor captures changes along fine edges and prominent pattern around eyes, eye brows, mouth, bulges and wrinkles of the face. The results of proposed technique are compared with different variants of LBP and LGC based techniques for both holistic and zoned images. Static Facial Expression in Wild (SFEW) dataset is selected for experimentation. Results clearly indicate that the suggested MBPC based technique surpasses other techniques with 96.5% and 67.2% accuracy for holistic and division based approach respectively. Moreover, results indicate that the performance of holistic approach is much higher than division based approach.
---
paper_title: Prototype-Based Discriminative Feature Learning for Kinship Verification

paper_content:

In this paper, we propose a new prototype-based discriminative feature learning (PDFL) method for kinship verification. Unlike most previous kinship verification methods which employ low-level hand-crafted descriptors such as local binary pattern and Gabor features for face representation, this paper aims to learn discriminative mid-level features to better characterize the kin relation of face images for kinship verification. To achieve this, we construct a set of face samples with unlabeled kin relation from the labeled face in the wild dataset as the reference set. Then, each sample in the training face kinship dataset is represented as a mid-level feature vector, where each entry is the corresponding decision value from one support vector machine hyperplane. Subsequently, we formulate an optimization function by minimizing the intraclass samples (with a kin relation) and maximizing the neighboring interclass samples (without a kin relation) with the mid-level features. To better use multiple low-level features for mid-level feature learning, we further propose a multiview PDFL method to learn multiple mid-level features to improve the verification performance. Experimental results on four publicly available kinship datasets show the superior performance of the proposed methods over both the state-of-the-art kinship verification methods and human ability in our kinship verification task.
---
paper_title: An Efficient Technique for Facial Expression Recognition Using Multistage Hidden Markov Model

paper_content:

Partition-based feature extraction is widely used in the pattern recognition and computer vision. This method is robust to some changes like occlusion, background, etc. In this paper, partition-based technique is used for feature extraction and extension of HMM is used as a classifier. The new introduced multistage HMM consists of two layers. In which bottom layer represents the atomic expression made by eyes, nose, and lips. Further upper layer represents the combination of these atomic expressions such as smile, fear, etc. Six basic facial expressions are recognized, i.e., anger, disgust, fear, joy, sadness, and surprise. Experimental result shows that proposed system performs better than normal HMM and has the overall accuracy of 85% using JAFFE database.
---
paper_title: Facial Feature Point Detection: A Comprehensive Survey

paper_content:

Abstract This paper presents a comprehensive survey of facial feature point detection with the assistance of abundant manually labeled images. Facial feature point detection favors many applications such as face recognition, animation, tracking, hallucination, expression analysis and 3D face modeling. Existing methods are categorized into two primary categories according to whether there is the need of a parametric shape model: parametric shape model-based methods and nonparametric shape model-based methods. Parametric shape model-based methods are further divided into two secondary classes according to their appearance models: local part model-based methods (e.g. constrained local model) and holistic model-based methods (e.g. active appearance model). Nonparametric shape model-based methods are divided into several groups according to their model construction process: exemplar-based methods, graphical model-based methods, cascaded regression-based methods, and deep learning based methods. Though significant progress has been made, facial feature point detection is still limited in its success by wild and real-world conditions: large variations across poses, expressions, illuminations, and occlusions. A comparative illustration and analysis of representative methods provides us a holistic understanding and deep insight into facial feature point detection, which also motivates us to further explore more promising future schemes.
---
paper_title: Biased subspace learning for misalignment-robust facial expression recognition

paper_content:

In this paper, we propose a biased subspace learning approach for misalignment-robust facial expression recognition. While a variety of facial expression recognition methods have been proposed in the literature, most of them only work well when face images are well registered and aligned. In many practical applications such as human robot interaction and visual surveillance, it is still challenging to obtain well-aligned face images for facial expression recognition due to currently imperfect computer vision techniques, especially under uncontrolled conditions. Motivated by the fact that interclass facial images with small differences are more easily mis-classified than those with large differences, we propose a biased linear discriminant analysis (BLDA) method by imposing large penalties on interclass samples with small differences and small penalties on those samples with large differences simultaneously, so that discriminative features can be better extracted for recognition. Moreover, we generate more virtually misaligned facial expression samples and assign different weights to them according to their occurrence probabilities in the testing phase to learn a weighted BLDA (WBLDA) feature space to extract misalignment-robust discriminative features for recognition. To better exploit the geometrical information of face samples, we propose a weighted biased margin Fisher analysis (WBMFA) method by employing a graph embedding criterion to extract discriminative information, so that the assumption of the Gaussian distribution of samples is not necessary. Experimental results on two widely used face databases are presented to show the efficacy of the proposed methods.
---
paper_title: Affect Detection: An Interdisciplinary Review of Models, Methods, and Their Applications

paper_content:

This survey describes recent progress in the field of Affective Computing (AC), with a focus on affect detection. Although many AC researchers have traditionally attempted to remain agnostic to the different emotion theories proposed by psychologists, the affective technologies being developed are rife with theoretical assumptions that impact their effectiveness. Hence, an informed and integrated examination of emotion theories from multiple areas will need to become part of computing practice if truly effective real-world systems are to be achieved. This survey discusses theoretical perspectives that view emotions as expressions, embodiments, outcomes of cognitive appraisal, social constructs, products of neural circuitry, and psychological interpretations of basic feelings. It provides meta-analyses on existing reviews of affect detection systems that focus on traditional affect detection modalities like physiology, face, and voice, and also reviews emerging research on more novel channels such as text, body language, and complex multimodal systems. This survey explicitly explores the multidisciplinary foundation that underlies all AC applications by describing how AC researchers have incorporated psychological theories of emotion and how these theories affect research questions, methods, results, and their interpretations. In this way, models and methods can be compared, and emerging insights from various disciplines can be more expertly integrated.
---
paper_title: A facial expression recognition system using robust face features from depth videos and deep learning

paper_content:

Abstract This work proposes a depth camera-based robust facial expression recognition (FER) system that can be adopted for better human machine interaction. Although video-based facial expression analysis has been focused on by many researchers, there are still various problems to be solved in this regard such as noise due to illumination variations over time. Depth video data in the helps to make an FER system person-independent as pixel values in depth images are distributed based on distances from a depth camera. Besides, depth images should resolve some privacy issues as real identity of a user can be hidden. The accuracy of an FER system is much dependent on the extraction of robust features. Here, we propose a novel method to extract salient features from depth faces that are further combined with deep learning for efficient training and recognition. Eight directional strengths are obtained for each pixel in a depth image where signs of some top strengths are arranged to represent unique as well as robust face features, which can be denoted as Modified Local Directional Patterns (MLDP). The MLDP features are further processed by Generalized Discriminant Analysis (GDA) for better face feature extraction. GDA is an efficient tool that helps distinguishing MLDP features of different facial expressions by clustering the features from the same expression as close as possible and separating the features from different expressions as much as possible in a non-linear space. Then, MLDP-GDA features are applied with Deep Belief Network (DBN) for training different facial expressions. Finally, the trained DBN is used to recognize facial expressions in a depth video for testing. The proposed approach was compared with other traditional approaches in a standalone system where the proposed one showed its superiority by achieving mean recognition rate of 96.25% where the other approaches could make 91.67% at the best. The deep learning-based training and recognition of the facial expression features can also be undertaken with cloud computing to support many users and make the system faster than a standalone system.
---
paper_title: Learning Expressionlets on Spatio-temporal Manifold for Dynamic Facial Expression Recognition

paper_content:

Facial expression is temporally dynamic event which can be decomposed into a set of muscle motions occurring in different facial regions over various time intervals. For dynamic expression recognition, two key issues, temporal alignment and semantics-aware dynamic representation, must be taken into account. In this paper, we attempt to solve both problems via manifold modeling of videos based on a novel mid-level representation, i.e. expressionlet. Specifically, our method contains three key components: 1) each expression video clip is modeled as a spatio-temporal manifold (STM) formed by dense low-level features, 2) a Universal Manifold Model (UMM) is learned over all low-level features and represented as a set of local ST modes to statistically unify all the STMs. 3) the local modes on each STM can be instantiated by fitting to UMM, and the corresponding expressionlet is constructed by modeling the variations in each local ST mode. With above strategy, expression videos are naturally aligned both spatially and temporally. To enhance the discriminative power, the expressionlet-based STM representation is further processed with discriminant embedding. Our method is evaluated on four public expression databases, CK+, MMI, Oulu-CASIA, and AFEW. In all cases, our method reports results better than the known state-of-the-art.
---
paper_title: A survey on facial recognition based on local directional and local binary patterns

paper_content:

The goal of this research project was to come up with a combined face recognition algorithm which out-performs existing algorithms on accuracy. The identification of individuals using face recognition techniques is a challenging task. This is due to the variations resulting from facial expressions, makeup, rotations, illuminations and gestures. Facial images also contain a great deal of redundant information, which negatively affects the performance of the recognition system. This paper proposes a novel approach for recognizing facial images from facial features using feature descriptors, namely local binary patterns(LBP) and local directional patterns(LDP). This research work consisted of three parts, namely face representation, feature extraction and classification. The useful and unique features of the facial images were extracted in the feature extraction phase. In classification, the face image was compared with the images from the database. The face area was divided into small regions from which local binary and directional patterns (LBP/LDP) histograms were extracted and concatenated into a single feature vector(histogram). Experiments performed on the Cohn-Kanade facial expression database obtained a good recognition rate 99% indicating superiority of the proposed method compared to other methods. The proposed included a combination of local binary pattern(LBP) and local directional patterns(LDP+LBP+Voting Classifier) as the feature extractor and voting classifier classification algorithm which is an aggregate classifier composed of k-Nearest Neighbor, decision trees and support vector machines. The results showed improved accuracy results as compared to other local binary pattern variants in both scenarios where small datasets or huge datasets were used.
---
paper_title: Facial Expression Recognition From Image Sequence Based on LBP and Taylor Expansion

paper_content:

The aim of an automatic video-based facial expression recognition system is to detect and classify human facial expressions from image sequence. An integrated automatic system often involves two components: 1) peak expression frame detection and 2) expression feature extraction. In comparison with the image-based expression recognition system, the video-based recognition system often performs online detection, which prefers low-dimensional feature representation for cost-effectiveness. Moreover, effective feature extraction is needed for classification. Many recent recognition systems often incorporate rich additional subjective information and thus become less efficient for real-time application. In our facial expression recognition system, first, we propose the double local binary pattern (DLBP) to detect the peak expression frame from the video. The proposed DLBP method has a much lower-dimensional size and can successfully reduce detection time. Besides, to handle the illumination variations in LBP, logarithm-laplace (LL) domain is further proposed to get a more robust facial feature for detection. Finally, the Taylor expansion theorem is employed in our system for the first time to extract facial expression feature. We propose the Taylor feature pattern (TFP) based on the LBP and Taylor expansion to obtain an effective facial feature from the Taylor feature map. Experimental results on the JAFFE and Cohn-Kanade data sets show that the proposed TFP method outperforms some state-of-the-art LBP-based feature extraction methods for facial expression feature extraction and can be suited for real-time applications.
---
paper_title: PCA Based Extracting Feature Using Fast Fourier Transform for Facial Expression Recognition

paper_content:

Facial expression recognition is prevalent in research area, and Principal Component Analysis (PCA) is a very common method in use. Noticing that few researches focus on pre-processing of images, which also enhances the results of PCA algorithm, we propose an improved approach of PCA based on facial expression recognition algorithm using Fast Fourier Transform (FFT), which combines amplitude spectrum of one image with phase spectrum of another image as a mixed image. Our experiments are based on Yale database and self-made image database. Testing and evaluating in several ways, the experimental results indicate our approach is effective.
---
paper_title: Facial expression recognition based on improved local binary pattern and class-regularized locality preserving projection

paper_content:

Abstract This paper provides a novel method for facial expression recognition, which distinguishes itself with the following two main contributions. First, an improved facial feature, called the expression-specific local binary pattern (es-LBP), is presented by emphasizing the partial information of human faces on particular fiducial points. Second, to enhance the connection between facial features and expression classes, class-regularized locality preserving projection (cr-LPP) is proposed, which aims at maximizing the class independence and simultaneously preserving the local feature similarity via dimensionality reduction. Simulation results show that the proposed approach is very effective for facial expression recognition.
---
paper_title: Emotion Recognition in the Wild via Convolutional Neural Networks and Mapped Binary Patterns

paper_content:

We present a novel method for classifying emotions from static facial images. Our approach leverages on the recent success of Convolutional Neural Networks (CNN) on face recognition problems. Unlike the settings often assumed there, far less labeled data is typically available for training emotion classification systems. Our method is therefore designed with the goal of simplifying the problem domain by removing confounding factors from the input images, with an emphasis on image illumination variations. This, in an effort to reduce the amount of data required to effectively train deep CNN models. To this end, we propose novel transformations of image intensities to 3D spaces, designed to be invariant to monotonic photometric transformations. These are applied to CASIA Webface images which are then used to train an ensemble of multiple architecture CNNs on multiple representations. Each model is then fine-tuned with limited emotion labeled training data to obtain final classification models. Our method was tested on the Emotion Recognition in the Wild Challenge (EmotiW 2015), Static Facial Expression Recognition sub-challenge (SFEW) and shown to provide a substantial, 15.36% improvement over baseline results (40% gain in performance).
---
paper_title: Illumination Invariant Facial Expression Recognition Using Selected Merged Binary Patterns for Real World Images

paper_content:

Abstract Automatic facial expression recognition has always been a challenging task to understand human behavior from real world images. Certain type of issues are associated with such images that include poor illumination, different orientations and varying pose. The proposed technique first applies Fast Fourier Transform and Contrast Limited Adaptive Histogram Equalization (FFT + CLAHE) method to compensate the poor illumination. Then merged binary pattern code (MBPC) is generated for every pixel. Two bits per neighbourhood are produced to form a 16-bit code per pixel. This code merges local features to improve the effectiveness of facial expression recognition system. MBPC descriptor captures changes along fine edges and prominent pattern around eyes, eye brows, mouth, bulges and wrinkles of the face. The results of proposed technique are compared with different variants of LBP and LGC based techniques for both holistic and zoned images. Static Facial Expression in Wild (SFEW) dataset is selected for experimentation. Results clearly indicate that the suggested MBPC based technique surpasses other techniques with 96.5% and 67.2% accuracy for holistic and division based approach respectively. Moreover, results indicate that the performance of holistic approach is much higher than division based approach.
---
paper_title: Performance Evaluation of PCA and ICA Algorithm for Facial Expression Recognition Application

paper_content:

In everyday interaction, our face is the basic and primary focus of attention. Out of many human psycho-signatures, the face provides a unique identification of a person by the virtue of its size, shape, and different expressions such as happy, sad, disgust, surprise, fear, anger, neutral, etc. In a human computer interaction, facial expression recognition is an interesting and one of the most challenging research areas. In the proposed work, principle component analysis (PCA) and independent component analysis (ICA) are used for the facial expressions recognition. Euclidean distance classifier and cosine similarity measure are used as the cost function for testing and verification of the images. Japanese Female Facial Expression (JAFFE) database and our own customized database are used for the analysis. The experimental result shows that ICA provides improved facial expression recognition in comparison with PCA. The PCA and ICA provides detection accuracy of 81.42 and 94.28 %, respectively.
---
paper_title: Illumination invariants in deep video expression recognition

paper_content:

Abstract In this paper we present architectures based on deep neural nets for expression recognition in videos, which are invariant to local scaling. We amalgamate autoencoder and predictor architectures using an adaptive weighting scheme coping with a reduced size labeled dataset, while enriching our models from enormous unlabeled sets. We further improve robustness to lighting conditions by introducing a new adaptive filter based on temporal local scale normalization. We provide superior results over known methods, including recent reported approaches based on neural nets.
---
paper_title: Facial Expression Recognition Utilizing Local Direction-Based Robust Features and Deep Belief Network

paper_content:

Emotional health plays very vital role to improve people’s quality of lives, especially for the elderly. Negative emotional states can lead to social or mental health problems. To cope with emotional health problems caused by negative emotions in daily life, we propose efficient facial expression recognition system to contribute in emotional healthcare system. Thus, facial expressions play a key role in our daily communications, and recent years have witnessed a great amount of research works for reliable facial expressions recognition (FER) systems. Therefore, facial expression evaluation or analysis from video information is very challenging and its accuracy depends on the extraction of robust features. In this paper, a unique feature extraction method is presented to extract distinguished features from the human face. For person independent expression recognition, depth video data is used as input to the system where in each frame, pixel intensities are distributed based on the distances to the camera. A novel robust feature extraction process is applied in this work which is named as local directional position pattern (LDPP). In LDPP, after extracting local directional strengths for each pixel such as applied in typical local directional pattern (LDP), top directional strength positions are considered in binary along with their strength sign bits. Considering top directional strength positions with strength signs in LDPP can differentiate edge pixels with bright as well as dark regions on their opposite sides by generating different patterns whereas typical LDP only considers directions representing the top strengths irrespective of their signs as well as position orders (i.e., directions with top strengths represent 1 and rest of them 0), which can generate the same patterns in this regard sometimes. Hence, LDP fails to distinguish edge pixels with opposite bright and dark regions in some cases which can be overcome by LDPP. Moreover, the LDPP capabilities are extended through principal component analysis (PCA) and generalized discriminant analysis (GDA) for better face characteristic illustration in expression. The proposed features are finally applied with deep belief network (DBN) for expression training and recognition.
---
paper_title: Facial Expression Recognition Based on Deep Evolutional Spatial-Temporal Networks

paper_content:

One key challenging issue of facial expression recognition is to capture the dynamic variation of facial physical structure from videos. In this paper, we propose a part-based hierarchical bidirectional recurrent neural network (PHRNN) to analyze the facial expression information of temporal sequences. Our PHRNN models facial morphological variations and dynamical evolution of expressions, which is effective to extract “temporal features” based on facial landmarks (geometry information) from consecutive frames. Meanwhile, in order to complement the still appearance information, a multi-signal convolutional neural network (MSCNN) is proposed to extract “spatial features” from still frames. We use both recognition and verification signals as supervision to calculate different loss functions, which are helpful to increase the variations of different expressions and reduce the differences among identical expressions. This deep evolutional spatial-temporal network (composed of PHRNN and MSCNN) extracts the partial-whole, geometry-appearance, and dynamic-still information, effectively boosting the performance of facial expression recognition. Experimental results show that this method largely outperforms the state-of-the-art ones. On three widely used facial expression databases (CK+, Oulu-CASIA, and MMI), our method reduces the error rates of the previous best ones by 45.5%, 25.8%, and 24.4%, respectively.
---
paper_title: Real-time pose invariant spontaneous smile detection using conditional random regression forests

paper_content:

Abstract Detecting spontaneous smile in unconstrained environment is a challenging problem mainly due to the large intra-class variations caused by head poses. This paper presents a real-time smile detection method based on conditional random regression forests. Since the relation between image patches and smile intensity is modelled conditional to head pose, the proposed smile detection method is not sensitive to head poses. To achieve high smile detection performance, techniques including regression forest, multiple-label dataset augmentation and non-informative patch removement are employed. Experimental results show that the proposed method achieves competitive performance to state-of-the-art deep neural network based methods on two challenging real-world datasets, although using hand-crafted features. A dynamical forest ensemble scheme is also presented to make a trade-off between smile detection performance and processing speed. In contrast to deep neural networks, the proposed method can run in real-time on general hardware without GPU.
---
paper_title: Facial Expression Recognition in Video with Multiple Feature Fusion

paper_content:

Video based facial expression recognition has been a long standing problem and attracted growing attention recently. The key to a successful facial expression recognition system is to exploit the potentials of audiovisual modalities and design robust features to effectively characterize the facial appearance and configuration changes caused by facial motions. We propose an effective framework to address this issue in this paper. In our study, both visual modalities (face images) and audio modalities (speech) are utilized. A new feature descriptor called Histogram of Oriented Gradients from Three Orthogonal Planes (HOG-TOP) is proposed to extract dynamic textures from video sequences to characterize facial appearance changes. And a new effective geometric feature derived from the warp transformation of facial landmarks is proposed to capture facial configuration changes. Moreover, the role of audio modalities on recognition is also explored in our study. We applied the multiple feature fusion to tackle the video-based facial expression recognition problems under lab-controlled environment and in the wild, respectively. Experiments conducted on the extended Cohn-Kanade (CK+) database and the Acted Facial Expression in Wild (AFEW) 4.0 database show that our approach is robust in dealing with video-based facial expression recognition problems under lab-controlled environment and in the wild compared with the other state-of-the-art methods.
---
paper_title: Collaborative discriminative multi-metric learning for facial expression recognition in video

paper_content:

Abstract Facial expression recognition in video has been an important and relatively new topic in human face analysis and attracted growing interests in recent years. Unlike conventional image-based facial expression recognition methods which recognize facial expression category from still images, facial expression recognition in video is more challenging because there are usually larger intra-class variations among facial frames within a video. This paper presents a collaborative discriminative multi-metric learning (CDMML) for facial expression recognition in video. We first compute multiple feature descriptors for each face video to describe facial appearance and motion information from different aspects. Then, we learn multiple distance metrics with these extracted multiple features collaboratively to exploit complementary and discriminative information for recognition. Experimental results on the Acted Facial Expression in Wild (AFEW) 4.0 and the extended Cohn–Kanada (CK+) datasets are presented to demonstrate the effectiveness of our proposed method.
---
paper_title: The Emotracker: Visualizing Contents, Gaze and Emotions at a Glance

paper_content:

In the last years, Affective Computing investigations have focused the efforts in the automatic extraction of human emotions and in increasing the success rates in the emotion recognition task. However, there is a lack of automatic tools that intuitively visualize the users' emotional information. In this work, we propose Emotracker, a novel tool based on the combination of eye tracking and facial emotional recognition technologies that allows the visualization of contents, user emotions and gaze at a glance, and offers a wide range of visualization options, including emotional saccade maps and emotional heat maps. The simultaneous consideration of gaze and emotion opens the door to the evaluation of user engagement, which is of great importance for content creators, but has also other promising fields of application in psychology research and, more specifically, in autism.
---
paper_title: What the face reveals : basic and applied studies of spontaneous expression using the facial action coding system (FACS)

paper_content:

Foreword Introduction The study of spontaneous facial expression in psychology I: BASIC RESEARCH ON EMOTION 1. Is the startle reaction an emotion? 2. The asymmetry of facial actions is inconsisten with models of hemispheric specialization 3. Coherence between expressive and experiential systmes in emotion 4. Will the real relationship between facial expression and affective experience please stand up: The case of exhilaration 5. Extraversion, alcohol, and enjoyment 6. Signs of appeasement: Evidence for the distinct displays of embarrassment, amusement, and shame 7. Genuine, suppressed, and faked facial behavior during exacerbation of chronic low back pain 8. The consistency of facial expressions of pain: a comparison across modalities 9. Smiles when lying 10. Behavioral markers and recognizability of the smile of enjoyment 11. Components and recognition of facial expression in the communication of emotion by actors 12. Differentiating emotion elicited and deliberate emotional facial expressions 13. Japanese and American infants' responses to arm restraint 14. Differential facial responses to four basic tests in newborns II: APPLIED RESEARCH 15. Facial expressions in affective disorders 16. Emotional experience and epxression in schizophrenia and depression 17. Interaction regulations used by schizophrenic and psychosomatic patients: Studies on facial behavior in dyadic interactions 18. Nonverbal expression of psychological states in psychiatric patients 19. Depression and suicide faces 20. Prototypical affective microsequences in psychotherapeutic interaction 21. Facial expressions of emotion and psychopathology in adolescent boys 22. Type A behavior pattern: Facial behavior and speech components Conclusion What we have learned by measuring the face Index
---
paper_title: Facial Expression Recognition in Video with Multiple Feature Fusion

paper_content:

Video based facial expression recognition has been a long standing problem and attracted growing attention recently. The key to a successful facial expression recognition system is to exploit the potentials of audiovisual modalities and design robust features to effectively characterize the facial appearance and configuration changes caused by facial motions. We propose an effective framework to address this issue in this paper. In our study, both visual modalities (face images) and audio modalities (speech) are utilized. A new feature descriptor called Histogram of Oriented Gradients from Three Orthogonal Planes (HOG-TOP) is proposed to extract dynamic textures from video sequences to characterize facial appearance changes. And a new effective geometric feature derived from the warp transformation of facial landmarks is proposed to capture facial configuration changes. Moreover, the role of audio modalities on recognition is also explored in our study. We applied the multiple feature fusion to tackle the video-based facial expression recognition problems under lab-controlled environment and in the wild, respectively. Experiments conducted on the extended Cohn-Kanade (CK+) database and the Acted Facial Expression in Wild (AFEW) 4.0 database show that our approach is robust in dealing with video-based facial expression recognition problems under lab-controlled environment and in the wild compared with the other state-of-the-art methods.
---
paper_title: Motion history for facial action detection in video

paper_content:

Enabling computer systems to recognize human facial expressions is a challenging research problem with many applications in behavioral science, medicine, security, and human-machine interaction. Instead of being another approach to automatic detection of prototypic facial expressions of emotion, this work attempts to analyze subtle changes in facial behavior by recognizing facial action units (AU, i.e. atomic facial signals) that produce expressions. This work proposes AU recognition based upon multilevel motion history images (MMHI), which can be seen as an extension to temporal templates introduced by Bobick and Davis. By recording motion history at multiple time Intervals (i.e., multilevel MHI) instead of recording it once for the entire image sequence, we overcome the problem of self-occlusion which is inherent to temporal templates original definition. For automatic classification of an input MMHI-represented face video in terms of 21 AU classes, two approaches are compared: a sparse network of Winnows (SNoW) and a standard kNearest neighbour (kNN) classifier. The system was tested on two different databases, the MMI-Face-DB developed by the authors and the Cohn-Kanade face database.
---
paper_title: Histograms of oriented gradients for human detection

paper_content:

We study the question of feature sets for robust visual object recognition; adopting linear SVM based human detection as a test case. After reviewing existing edge and gradient based descriptors, we show experimentally that grids of histograms of oriented gradient (HOG) descriptors significantly outperform existing feature sets for human detection. We study the influence of each stage of the computation on performance, concluding that fine-scale gradients, fine orientation binning, relatively coarse spatial binning, and high-quality local contrast normalization in overlapping descriptor blocks are all important for good results. The new approach gives near-perfect separation on the original MIT pedestrian database, so we introduce a more challenging dataset containing over 1800 annotated human images with a large range of pose variations and backgrounds.
---
paper_title: Collaborative discriminative multi-metric learning for facial expression recognition in video

paper_content:

Abstract Facial expression recognition in video has been an important and relatively new topic in human face analysis and attracted growing interests in recent years. Unlike conventional image-based facial expression recognition methods which recognize facial expression category from still images, facial expression recognition in video is more challenging because there are usually larger intra-class variations among facial frames within a video. This paper presents a collaborative discriminative multi-metric learning (CDMML) for facial expression recognition in video. We first compute multiple feature descriptors for each face video to describe facial appearance and motion information from different aspects. Then, we learn multiple distance metrics with these extracted multiple features collaboratively to exploit complementary and discriminative information for recognition. Experimental results on the Acted Facial Expression in Wild (AFEW) 4.0 and the extended Cohn–Kanada (CK+) datasets are presented to demonstrate the effectiveness of our proposed method.
---
paper_title: ECG-Based Concentration Recognition With Multi-Task Regression

paper_content:

Objective: Recognition of human activities and mental states using wearable sensors and smartphones has attracted considerable attention recently. In particular, prediction of the stress level of a subject using an electrocardiogram sensor has been studied extensively. In this paper, we attempt to predict the degree of concentration by using heart-rate features. However, due to strong diversity in individuals and high sampling costs, building an accurate prediction model is still highly challenging. Method: To overcome these difficulties, we propose to use a multitask learning (MTL) technique for effectively sharing information among similar individuals. Result: Through experiments with 18 healthy subjects performing daily office works, such as writing reports, we demonstrate that the proposed method significantly improves the accuracy of concentration prediction in small sample situations. Conclusion: The performance of the MTL method is shown to be stable across different subjects, which is an important advantage over conventional models. Significance: This improvement has significant impact in real-world concentration recognition because the data collection burden of each user can be drastically mitigated.
---
paper_title: Fusion of facial expressions and EEG for implicit affective tagging

paper_content:

The explosion of user-generated, untagged multimedia data in recent years, generates a strong need for efficient search and retrieval of this data. The predominant method for content-based tagging is through slow, labor-intensive manual annotation. Consequently, automatic tagging is currently a subject of intensive research. However, it is clear that the process will not be fully automated in the foreseeable future. We propose to involve the user and investigate methods for implicit tagging, wherein users' responses to the interaction with the multimedia content are analyzed in order to generate descriptive tags. Here, we present a multi-modal approach that analyses both facial expressions and electroencephalography (EEG) signals for the generation of affective tags. We perform classification and regression in the valence-arousal space and present results for both feature-level and decision-level fusion. We demonstrate improvement in the results when using both modalities, suggesting the modalities contain complementary information.
---
paper_title: Emotional State Recognition with EEG Signals Using Subject Independent Approach

paper_content:

EEG signals vary from human to human and hence it is very difficult to create a subject independent emotion recognition system. Even though subject dependent methodologies could achieve good emotion recognition accuracy, the subject-independent approaches are still in infancy. EEG is reliable than facial expression or speech signal to recognize emotions, since it can not be fake. In this paper, a Multilayer Perceptron neural network based subject-independent emotion recognition system is proposed. Performance evaluation of the proposed system, on the benchmark DEAP dataset shows good accuracy compared to the state of the art subject independent methods.
---
paper_title: Towards unravelling the relationship between on-body, environmental and emotion data using sensor information fusion approach

paper_content:

Abstract Over the past few years, there has been a noticeable advancement in environmental models and information fusion systems taking advantage of the recent developments in sensor and mobile technologies. However, little attention has been paid so far to quantifying the relationship between environment changes and their impact on our bodies in real-life settings. In this paper, we identify a data driven approach based on direct and continuous sensor data to assess the impact of the surrounding environment and physiological changes and emotion. We aim at investigating the potential of fusing on-body physiological signals, environmental sensory data and on-line self-report emotion measures in order to achieve the following objectives: (1) model the short term impact of the ambient environment on human body, (2) predict emotions based on-body sensors and environmental data. To achieve this, we have conducted a real-world study ‘in the wild’ with on-body and mobile sensors. Data was collected from participants walking around Nottingham city centre, in order to develop analytical and predictive models. Multiple regression, after allowing for possible confounders, showed a noticeable correlation between noise exposure and heart rate. Similarly, UV and environmental noise have been shown to have a noticeable effect on changes in ElectroDermal Activity (EDA ). Air pressure demonstrated the greatest contribution towards the detected changes in body temperature and motion. Also, significant correlation was found between air pressure and heart rate . Finally, decision fusion of the classification results from different modalities is performed. To the best of our knowledge this work presents the first attempt at fusing and modelling data from environmental and physiological sources collected from sensors in a real-world setting.
---
paper_title: Wearable EOG goggles: Seamless sensing and context-awareness in everyday environments

paper_content:

In this article we introduce the analysis of eye motion as a new input modality for activity recognition, context-awareness and mobile HCI applications. We describe a novel embedded eye tracker that, in contrast to common systems using video cameras, relies on Electrooculography (EOG). This self-contained wearable device consists of goggles with dry electrodes integrated into the frame and a small pocket-worn component with a DSP for real-time EOG signal processing. It can store data locally for long-term recordings or stream processed EOG signals to a remote device over Bluetooth. We show how challenges associated with wearability, eye motion analysis and signal artefacts caused by physical activity can be addressed with a combination of a special mechanical design, optimised algorithms for eye movement detection and adaptive signal processing. In two case studies, we demonstrate that EOG is a suitable measurement technique for the recognition of reading activity and eye-based human-computer interaction. Eventually, wearable EOG goggles may pave the way for seamless eye movement analysis in everyday environments and new forms of context-awareness not possible today.
---
paper_title: Evolutionary computation algorithms for feature selection of EEG-based emotion recognition using mobile sensors

paper_content:

A review of state-of-the-art feature extraction methods from electroencephalogram signals.A new framework using evolutionary algorithms to find the most optimal features set and channels.Comprehensive experimental results based on two public datasets and one newly collected dataset. There is currently no standard or widely accepted subset of features to effectively classify different emotions based on electroencephalogram (EEG) signals. While combining all possible EEG features may improve the classification performance, it can lead to high dimensionality and worse performance due to redundancy and inefficiency. To solve the high-dimensionality problem, this paper proposes a new framework to automatically search for the optimal subset of EEG features using evolutionary computation (EC) algorithms. The proposed framework has been extensively evaluated using two public datasets (MAHNOB, DEAP) and a new dataset acquired with a mobile EEG sensor. The results confirm that EC algorithms can effectively support feature selection to identify the best EEG features and the best channels to maximize performance over a four-quadrant emotion classification problem. These findings are significant for informing future development of EEG-based emotion classification because low-cost mobile EEG sensors with fewer electrodes are becoming popular for many new applications.
---
paper_title: A facial expression recognition system using robust face features from depth videos and deep learning

paper_content:

Abstract This work proposes a depth camera-based robust facial expression recognition (FER) system that can be adopted for better human machine interaction. Although video-based facial expression analysis has been focused on by many researchers, there are still various problems to be solved in this regard such as noise due to illumination variations over time. Depth video data in the helps to make an FER system person-independent as pixel values in depth images are distributed based on distances from a depth camera. Besides, depth images should resolve some privacy issues as real identity of a user can be hidden. The accuracy of an FER system is much dependent on the extraction of robust features. Here, we propose a novel method to extract salient features from depth faces that are further combined with deep learning for efficient training and recognition. Eight directional strengths are obtained for each pixel in a depth image where signs of some top strengths are arranged to represent unique as well as robust face features, which can be denoted as Modified Local Directional Patterns (MLDP). The MLDP features are further processed by Generalized Discriminant Analysis (GDA) for better face feature extraction. GDA is an efficient tool that helps distinguishing MLDP features of different facial expressions by clustering the features from the same expression as close as possible and separating the features from different expressions as much as possible in a non-linear space. Then, MLDP-GDA features are applied with Deep Belief Network (DBN) for training different facial expressions. Finally, the trained DBN is used to recognize facial expressions in a depth video for testing. The proposed approach was compared with other traditional approaches in a standalone system where the proposed one showed its superiority by achieving mean recognition rate of 96.25% where the other approaches could make 91.67% at the best. The deep learning-based training and recognition of the facial expression features can also be undertaken with cloud computing to support many users and make the system faster than a standalone system.
---
paper_title: Dense 3D facial reconstruction from a single depth image in unconstrained environment

paper_content:

With the increasing demands of applications in virtual reality such as 3D films, virtual human–machine interactions and virtual agents, the analysis of 3D human face is considered to be more and more important as a fundamental step in these tasks. Due to information provided by the additional dimension, 3D facial reconstruction enables aforementioned tasks to be achieved with higher accuracy than those based on 2D facial analysis. The denser the 3D facial model is, the more information it could provide. However, most existing dense 3D facial reconstruction methods require complicated processing and high system cost. To this end, this paper presents a novel method that simplifies the process of dense 3D facial reconstruction by employing only one frame of depth data obtained with an off-the-shelf RGB-D sensor. The proposed method is composed of two main stages: (a) the acquisition of the initial 3D facial point cloud with automatically 3D facial region cropping, and (b) the generating of the dense facial point cloud with RBF-based adaptive 3D point interpolation. Experiments reported in this paper demonstrate the competitive results with real-world data.
---
paper_title: Human action recognition using depth motion maps pyramid and discriminative collaborative representation classifier

paper_content:

Human action recognition is a challenging task in machine learning and pattern recognition. This paper presents an action recognition framework based on depth sequences. An effective feature descriptor named depth motion maps pyramid (DMMP) inspired by DMMs is developed. First, a series of DMMs with temporal scales are constructed to effectively capture spatial–temporal motion patterns of human actions. Then these DMMs are fused to obtain the final descriptor named DMMs pyramid. Second, we propose a discriminative collaborative representation classifier (DCRC), where an extra constraint on the collaborative coefficient is imposed to provide prior knowledge for the representation coefficient. In addition, we apply DCRC to encode the obtained features and recognize the human actions. The proposed framework is evaluated on MSR three-dimensional (3-D) action datasets, MSR hand gesture dataset, UTD-MHAD, and MSR daily Activity3D dataset, respectively. The experimental results indicate the effectiveness of our proposed method for human action recognition.
---
paper_title: Robust facial expression recognition using RGB-D images and multichannel features

paper_content:

Traditional and classical methods of facial expression recognition are mainly based on intensity image and are prone to be disturbed by illumination, poses, and disguise. This research aims to develop a robust facial expression recognition method using RGB-D images and multichannel features. Based on image entropy and visual saliency, facial texture features are firstly extracted from RGB images and depth images to construct the Histogram of Oriented Gradient (HOG) descriptors. And then, we extract geometric features of RGB images using Active Appearance Model (AAM). Combining the HOG texture features with the AAM geometric feature, we build a robust multichannel feature vector for facial expression recognition. On this basis, an improved Support Vector Machine (SVM) algorithm, namely GS-SVM, is used to classify facial expression recognition. The proposed GS-SVM algorithm applies Grid Search method to optimize the best parameters for SVM classifier and estimate the accuracy of each parameter combination in specified range. Finally, the proposed methods are tested and evaluated on the merged RGB-D database. Experimental results show that the proposed algorithm not only achieves a higher average recognition rate but also is robust to uncontrolled environments.
---
paper_title: Facial Expression Recognition Utilizing Local Direction-Based Robust Features and Deep Belief Network

paper_content:

Emotional health plays very vital role to improve people’s quality of lives, especially for the elderly. Negative emotional states can lead to social or mental health problems. To cope with emotional health problems caused by negative emotions in daily life, we propose efficient facial expression recognition system to contribute in emotional healthcare system. Thus, facial expressions play a key role in our daily communications, and recent years have witnessed a great amount of research works for reliable facial expressions recognition (FER) systems. Therefore, facial expression evaluation or analysis from video information is very challenging and its accuracy depends on the extraction of robust features. In this paper, a unique feature extraction method is presented to extract distinguished features from the human face. For person independent expression recognition, depth video data is used as input to the system where in each frame, pixel intensities are distributed based on the distances to the camera. A novel robust feature extraction process is applied in this work which is named as local directional position pattern (LDPP). In LDPP, after extracting local directional strengths for each pixel such as applied in typical local directional pattern (LDP), top directional strength positions are considered in binary along with their strength sign bits. Considering top directional strength positions with strength signs in LDPP can differentiate edge pixels with bright as well as dark regions on their opposite sides by generating different patterns whereas typical LDP only considers directions representing the top strengths irrespective of their signs as well as position orders (i.e., directions with top strengths represent 1 and rest of them 0), which can generate the same patterns in this regard sometimes. Hence, LDP fails to distinguish edge pixels with opposite bright and dark regions in some cases which can be overcome by LDPP. Moreover, the LDPP capabilities are extended through principal component analysis (PCA) and generalized discriminant analysis (GDA) for better face characteristic illustration in expression. The proposed features are finally applied with deep belief network (DBN) for expression training and recognition.
---
paper_title: Facial expression recognition using depth information and spatiotemporal features

paper_content:

Facial expression recognition is considered to be one of the very important topics in image processing, pattern recognition, and computer vision due to its broad applications such as human-computer interaction, behavior analysis, and image understanding. In this work, a novel approach is proposed to recognize some facial expressions from time-sequential depth videos. First of all, efficient Local Directional Pattern (LDP) features are obtained from the time-sequential depth faces that are further augmented with the optical flow features. The augmented features are classified by Generalized Discriminant Analysis (GDA) to make the features more robust and finally, the spatiotemporal features are fed into Hidden Markov Models (HMMs) to train and recognize different facial expressions successfully. The depth information-based proposed facial expression recognition approach is compared to the conventional approaches where the proposed one outperforms others.
---
paper_title: Presentation Attack Detection for Face Recognition Using Light Field Camera

paper_content:

The vulnerability of face recognition systems is a growing concern that has drawn the interest from both academic and research communities. Despite the availability of a broad range of face presentation attack detection (PAD) (or countermeasure or antispoofing) schemes, there exists no superior PAD technique due to evolution of sophisticated presentation attacks (or spoof attacks). In this paper, we present a new perspective for face presentation attack detection by introducing light field camera (LFC). Since the use of a LFC can record the direction of each incoming ray in addition to the intensity, it exhibits an unique characteristic of rendering multiple depth (or focus) images in a single capture. Thus, we present a novel approach that involves exploring the variation of the focus between multiple depth (or focus) images rendered by the LFC that in turn can be used to reveal the presentation attacks. To this extent, we first collect a new face artefact database using LFC that comprises of 80 subjects. Face artefacts are generated by simulating two widely used attacks, such as photo print and electronic screen attack. Extensive experiments carried out on the light field face artefact database have revealed the outstanding performance of the proposed PAD scheme when benchmarked with various well established state-of-the-art schemes.
---
paper_title: Survey on RGB, 3D, Thermal, and Multimodal Approaches for Facial Expression Recognition: History, Trends, and Affect-Related Applications

paper_content:

Facial expressions are an important way through which humans interact socially. Building a system capable of automatically recognizing facial expressions from images and video has been an intense field of study in recent years. Interpreting such expressions remains challenging and much research is needed about the way they relate to human affect. This paper presents a general overview of automatic RGB, 3D, thermal and multimodal facial expression analysis. We define a new taxonomy for the field, encompassing all steps from face detection to facial expression recognition, and describe and classify the state of the art methods accordingly. We also present the important datasets and the bench-marking of most influential methods. We conclude with a general discussion about trends, important questions and future lines of research.
---
paper_title: Thermal image analysis for polygraph testing

paper_content:

We have designed, developed, and tested a very promising thermal image analysis method for polygraph testing. The method achieved a correct classification rate of CCR= 84% on the test population to our avail. This method, once refined, can serve as an additional channel for increasing the reliability and accuracy of traditional polygraph examination. We extract subtle facial temperature fluctuation patterns through nonlinear heat transfer modeling. The modeling transforms raw thermal data to blood flow rate information. Then, we use the slope of the average periorbital blood flow rate as the feature of a binary classification scheme. The results come to support our previous laboratory findings about the importance of periorbital blood flow in anxious states.
---
paper_title: Visual Learning of Texture Descriptors for Facial Expression Recognition in Thermal Imagery

paper_content:

Facial expression recognition is an active research area that finds a potential application in human emotion analysis. This work presents an illumination independent approach for facial expression recognition based on long wave infrared imagery. In general, facial expression recognition systems are designed considering the visible spectrum. This makes the recognition process not robust enough to be deployed in poorly illuminated environments. Common approaches to facial expression recognition of static images are designed considering three main parts: (1) region of interest selection, (2) feature extraction, and (3) image classification. Most published articles propose methodologies that solve each of these tasks in a decoupled way. We propose a Visual Learning approach based on evolutionary computation that solves the first two tasks simultaneously using a single evolving process. The first task consists in the selection of a set of suitable regions where the feature extraction is performed. The second task consists in tuning the parameters that defines the extraction of the Gray Level Co-occurrence Matrix used to compute region descriptors, as well as the selection of the best subsets of descriptors. The output of these two tasks is used for classification by a SVM committee. A dataset of thermal images with three different expression classes is used to validate the performance. Experimental results show effective classification when compared to a human observer, as well as a PCA-SVM approach. This paper concludes that: (1) thermal Imagery provides relevant information for FER, and (2) that the developed methodology can be taken as an efficient learning mechanism for different types of pattern recognition problems.
---
paper_title: Automated Facial Expression Classification and affect interpretation using infrared measurement of facial skin temperature variations

paper_content:

Machines would require the ability to perceive and adapt to affects for achieving artificial sociability. Most autonomous systems use Automated Facial Expression Classification (AFEC) and Automated Affect Interpretation (AAI) to achieve sociability. Varying lighting conditions, occlusion, and control over physiognomy can influence the real life performance of vision-based AFEC systems. Physiological signals provide complementary information for AFEC and AAI. We employed transient facial thermal features for AFEC and AAI. Infrared thermal images with participants' normal expression and intentional expressions of happiness, sadness, disgust, and fear were captured. Facial points that undergo significant thermal changes with a change in expression termed as Facial Thermal Feature Points (FTFPs) were identified. Discriminant analysis was invoked on principal components derived from the Thermal Intensity Values (TIVs) recorded at the FTFPs. The cross-validation and person-independent classification respectively resulted in 66.28p and 56.0p success rates. Classification significance tests suggest that (1) like other physiological cues, facial skin temperature also provides useful information about affective states and their facial expression; (2) patterns of facial skin temperature variation can complement other cues for AFEC and AAI; and (3) infrared thermal imaging may help achieve artificial sociability in robots and autonomous systems.
---
paper_title: Facial expression recognition for speaker using thermal image processing and speech recognition system

paper_content:

I investigated a method for facial expression recognition for a human speaker by using thermal image processing and a speech recognition system. In this study, we improved our speech recognition system to save thermal images at the three timing positions of just before speaking, and just when speaking the phonemes of the first and last vowels. With this method, intentional facial expressions of "angry", "happy", "neutral", "sad", and "surprised" were discriminable with good recognition accuracy.
---
paper_title: Analysis of facial thermal variations in response to emotion: eliciting film clips

paper_content:

NVESD is currently researching relationships between facial thermal signatures and underlying physiological stress. ::: This paper describes some of the initial findings of a study involving participants watching emotion-eliciting video clips ::: while being monitored by a high-resolution, Mid-Wave InfraRed (MWIR), cooled sensor system. Both global and region-specific temperature profiles of the face were obtained for each participant from thermal videos recorded during the experiment. Changes in the global temperature patterns of each participant were shown to be consistent with changes in the visual and auditory stimuli. In some cases, there were inconsistencies in thermal trends among local facial regions ::: of individual participants. This indicates the possibility that different regions may exhibit different thermal patterns in ::: response to a stimulus.
---
paper_title: Facial Expression Recognition from Infrared Thermal Videos

paper_content:

In this paper, a spontaneous facial expression recognition method using infrared thermal videos is proposed. Firstly, the sequence features are extracted from the infrared thermal horizontal and vertical temperature difference sequences of different facial sub-regions. Secondly, a feature subset is selected according to their F-values. Thirdly, the Adaboost algorithm, with the weak classifiers of k-Nearest Neighbor, is used to classify facial expressions in arousal and valence dimensions. Finally, experiments on the Natural Visible and Infrared facial Expression (USTC-NVIE) database demonstrates the effectiveness of our approach.
---
paper_title: Classifying pretended and evoked facial expressions of positive and negative affective states using infrared measurement of skin temperature

paper_content:

Earlier researchers were able to extract the transient facial thermal features from thermal infrared images (TIRIs) to make binary distinctions between the expressions of affective states. However, effective human-computer interaction would require machines to distinguish between the subtle facial expressions of affective states. This work, for the first time, attempts to use the transient facial thermal features for recognizing a much wider range of facial expressions. A database of 324 time-sequential, visible-spectrum, and thermal facial images was developed representing different facial expressions from 23 participants in different situations. A novel facial thermal feature extraction, selection, and classification approach was developed and invoked on various Gaussian mixture models constructed using: neutral and pretended happy and sad faces, faces with multiple positive and negative facial expressions, faces with neutral and six (pretended) basic facial expressions, and faces with evoked happiness, sadness, disgust, and anger. This work demonstrates that (1) infrared imaging can be used to observe the affective-state-specific facial thermal variations, (2) pixel-grey level analysis of TIRIs can help localise significant facial thermal feature points along the major facial muscles, and (3) cluster-analytic classification of transient thermal features can help distinguish between the facial expressions of affective states in an optimized eigenspace of input thermal feature vectors. The observed classification results exhibited influence of a Gaussian mixture model's structure on classifier-performance. The work also unveiled some pertinent aspects of future research on the use of facial thermal features in automated facial expression classification and affect recognition.
---
paper_title: Spontaneous Facial Expression Recognition by Fusing Thermal Infrared and Visible Images

paper_content:

In this paper, we propose a method for spontaneous facial expression recognition by fusing features extracted from visible and thermal infrared images. First, the active appearance model parameters and head motion features are extracted from the visible images, and several thermal statistical parameters are extracted from the infrared images. Second, a multiple genetic algorithms-based fusion method is proposed for fusing these two spectrums. We use this proposed fusion method to search for the optimal combination of a similarity measurement and a feature subset. Then, a k-nearest neighbors classifier with the optimal combination is used to classify spontaneous facial expressions. Comparative experiments implemented on the Natural Visible and Infrared Facial Expression database show the effectiveness of the proposed similarity measurement and the feature selection method, and demonstrate the fusion method’s advantage over only using visible features.
---
paper_title: Automatic Feature Localization in Thermal Images for Facial Expression Recognition

paper_content:

We propose an unsupervised Local and Global feature extraction paradigm to approach the problem of facial expression recognition in thermal images. Starting from local, low-level features computed at interest point locations, our approach combines the localization of facial features with the holistic approach. The detailed steps are as follows: First, face localization using bi-modal thresholding is accomplished in order to localize facial features by way of a novel interest point detection and clustering approach. Second, we compute representative Eigenfeatures for feature extraction. Third, facial expression classification is made with a Support Vector Machine Committiee. Finally, the experiments over the IRIS data-set show that automation was achieved with good feature localization and classification performance.
---
paper_title: Posed and spontaneous expression recognition through modeling their spatial patterns

paper_content:

This paper presents a new method to recognize posed and spontaneous expressions through modeling their spatial patterns. Gender and expression categories are employed as privileged information to further improve the recognition. The proposed approach includes three steps. First, geometric features about facial shape and Action Unit variations are extracted from the differences between apex and onset facial images to capture the spatial facial variation. Second, statistical hypothesis testings are conducted to explore the differences between posed and spontaneous expressions using the defined geometric features from three aspects: all samples, samples given the gender information, and samples given expression categories. Third, several Bayesian networks are built to capture posed and spontaneous spatial facial patterns respectively given gender and expression categories. The statistical analysis results on the USTC-NVIE and SPOS databases both demonstrate the effectiveness of the proposed geometric features. The recognition results on the USTC-NVIE database indicate that the privileged information of gender and expression can help model the spatial patterns caused by posed and spontaneous expressions. The recognition results on both databases outperform those of the state of the art.
---
paper_title: Offline Deformable Face Tracking in Arbitrary Videos

paper_content:

Generic face detection and facial landmark localization in static imagery are among the most mature and well-studied problems in machine learning and computer vision. Currently, the top performing face detectors achieve a true positive rate of around 75-80% whilst maintaining low false positive rates. Furthermore, the top performing facial landmark localization algorithms obtain low point-to-point errors for more than 70% of commonly benchmarked images captured under unconstrained conditions. The task of facial landmark tracking in videos, however, has attracted much less attention. Generally, a tracking-by-detection framework is applied, where face detection and landmark localization are employed in every frame in order to avoid drifting. Thus, this solution is equivalent to landmark detection in static imagery. Empirically, a straightforward application of such a framework cannot achieve higher performance, on average, than the one reported for static imagery. In this paper, we show for the first time, to the best of our knowledge, that the results of generic face detection and landmark localization can be used to recursively train powerful and accurate person-specific face detectors and landmark localization methods for offline deformable tracking. The proposed pipeline can track landmarks in very challenging long-term sequences captured under arbitrary conditions. The pipeline was used as a semi-automatic tool to annotate the majority of the videos of the 300-VW Challenge.
---
paper_title: Localizing parts of faces using a consensus of exemplars

paper_content:

We present a novel approach to localizing parts in images of human faces. The approach combines the output of local detectors with a non-parametric set of global models for the part locations based on over one thousand hand-labeled exemplar images. By assuming that the global models generate the part locations as hidden variables, we derive a Bayesian objective function. This function is optimized using a consensus of models for these hidden variables. The resulting localizer handles a much wider range of expression, pose, lighting and occlusion than prior ones. We show excellent performance on a new dataset gathered from the internet and show that our localizer achieves state-of-the-art performance on the less challenging BioID dataset.
---
paper_title: Learning Social Relation Traits from Face Images

paper_content:

Social relation defines the association, e.g, warm, friendliness, and dominance, between two or more people. Motivated by psychological studies, we investigate if such fine-grained and high-level relation traits can be characterised and quantified from face images in the wild. To address this challenging problem we propose a deep model that learns a rich face representation to capture gender, expression, head pose, and age-related attributes, and then performs pairwise-face reasoning for relation prediction. To learn from heterogeneous attribute sources, we formulate a new network architecture with a bridging layer to leverage the inherent correspondences among these datasets. It can also cope with missing target attribute labels. Extensive experiments show that our approach is effective for fine-grained social relation learning in images and videos.
---
paper_title: The First Facial Landmark Tracking in-the-Wild Challenge: Benchmark and Results

paper_content:

Detection and tracking of faces in image sequences is among the most well studied problems in the intersection of statistical machine learning and computer vision. Often, tracking and detection methodologies use a rigid representation to describe the facial region 1, hence they can neither capture nor exploit the non-rigid facial deformations, which are crucial for countless of applications (e.g., facial expression analysis, facial motion capture, high-performance face recognition etc.). Usually, the non-rigid deformations are captured by locating and tracking the position of a set of fiducial facial landmarks (e.g., eyes, nose, mouth etc.). Recently, we witnessed a burst of research in automatic facial landmark localisation in static imagery. This is partly attributed to the availability of large amount of annotated data, many of which have been provided by the first facial landmark localisation challenge (also known as 300-W challenge). Even though now well established benchmarks exist for facial landmark localisation in static imagery, to the best of our knowledge, there is no established benchmark for assessing the performance of facial landmark tracking methodologies, containing an adequate number of annotated face videos. In conjunction with ICCV'2015 we run the first competition/challenge on facial landmark tracking in long-term videos. In this paper, we present the first benchmark for long-term facial landmark tracking, containing currently over 110 annotated videos, and we summarise the results of the competition.
---
paper_title: CAS(ME)$^2$ : A Database for Spontaneous Macro-Expression and Micro-Expression Spotting and Recognition

paper_content:

Deception is a very common phenomenon and its detection can be beneficial to our daily lives. Compared with other deception cues, micro-expression has shown great potential as a promising cue for deception detection. The spotting and recognition of micro-expression from long videos may significantly aid both law enforcement officers and researchers. However, database that contains both micro-expression and macro-expression in long videos is still not publicly available. To facilitate development in this field, we present a new database, Chinese Academy of Sciences Macro-Expressions and Micro-Expressions (CAS(ME) $^2$ ), which provides both macro-expressions and micro-expressions in two parts (A and B). Part A contains 87 long videos that contain spontaneous macro-expressions and micro-expressions. Part B includes 300 cropped spontaneous macro-expression samples and 57 micro-expression samples. The emotion labels are based on a combination of action units (AUs), self-reported emotion for every facial movement, and the emotion types of emotion-evoking videos. Local Binary Pattern (LBP) was employed for the spotting and recognition of macro-expressions and micro-expressions and the results were reported as a baseline evaluation. The CAS(ME) $^2$ database offers both long videos and cropped expression samples, which may aid researchers in developing efficient algorithms for the spotting and recognition of macro-expressions and micro-expressions.
---
paper_title: Facial Expression Recognition in Video with Multiple Feature Fusion

paper_content:

Video based facial expression recognition has been a long standing problem and attracted growing attention recently. The key to a successful facial expression recognition system is to exploit the potentials of audiovisual modalities and design robust features to effectively characterize the facial appearance and configuration changes caused by facial motions. We propose an effective framework to address this issue in this paper. In our study, both visual modalities (face images) and audio modalities (speech) are utilized. A new feature descriptor called Histogram of Oriented Gradients from Three Orthogonal Planes (HOG-TOP) is proposed to extract dynamic textures from video sequences to characterize facial appearance changes. And a new effective geometric feature derived from the warp transformation of facial landmarks is proposed to capture facial configuration changes. Moreover, the role of audio modalities on recognition is also explored in our study. We applied the multiple feature fusion to tackle the video-based facial expression recognition problems under lab-controlled environment and in the wild, respectively. Experiments conducted on the extended Cohn-Kanade (CK+) database and the Acted Facial Expression in Wild (AFEW) 4.0 database show that our approach is robust in dealing with video-based facial expression recognition problems under lab-controlled environment and in the wild compared with the other state-of-the-art methods.
---
paper_title: Static facial expression analysis in tough conditions: Data, evaluation protocol and benchmark

paper_content:

Quality data recorded in varied realistic environments is vital for effective human face related research. Currently available datasets for human facial expression analysis have been generated in highly controlled lab environments. We present a new static facial expression database Static Facial Expressions in the Wild (SFEW) extracted from a temporal facial expressions database Acted Facial Expressions in the Wild (AFEW) [9], which we have extracted from movies. In the past, many robust methods have been reported in the literature. However, these methods have been experimented on different databases or using different protocols within the same databases. The lack of a standard protocol makes it difficult to compare systems and acts as a hindrance in the progress of the field. Therefore, we propose a person independent training and testing protocol for expression recognition as part of the BEFIT workshop. Further, we compare our dataset with the JAFFE and Multi-PIE datasets and provide baseline results.
---
paper_title: The Extended Cohn-Kanade Dataset (CK+): A complete dataset for action unit and emotion-specified expression

paper_content:

In 2000, the Cohn-Kanade (CK) database was released for the purpose of promoting research into automatically detecting individual facial expressions. Since then, the CK database has become one of the most widely used test-beds for algorithm development and evaluation. During this period, three limitations have become apparent: 1) While AU codes are well validated, emotion labels are not, as they refer to what was requested rather than what was actually performed, 2) The lack of a common performance metric against which to evaluate new algorithms, and 3) Standard protocols for common databases have not emerged. As a consequence, the CK database has been used for both AU and emotion detection (even though labels for the latter have not been validated), comparison with benchmark algorithms is missing, and use of random subsets of the original database makes meta-analyses difficult. To address these and other concerns, we present the Extended Cohn-Kanade (CK+) database. The number of sequences is increased by 22% and the number of subjects by 27%. The target expression for each sequence is fully FACS coded and emotion labels have been revised and validated. In addition to this, non-posed sequences for several types of smiles and their associated metadata have been added. We present baseline results using Active Appearance Models (AAMs) and a linear support vector machine (SVM) classifier using a leave-one-out subject cross-validation for both AU and emotion detection for the posed data. The emotion and AU labels, along with the extended image data and tracked landmarks will be made available July 2010.
---
paper_title: The CMU Pose, Illumination, and Expression (PIE) database

paper_content:

Between October 2000 and December 2000, we collected a database of over 40,000 facial images of 68 people. Using the CMU (Carnegie Mellon University) 3D Room, we imaged each person across 13 different poses, under 43 different illumination conditions, and with four different expressions. We call this database the CMU Pose, Illumination and Expression (PIE) database. In this paper, we describe the imaging hardware, the collection procedure, the organization of the database, several potential uses of the database, and how to obtain the database.
---
paper_title: Multi-Velocity Neural Networks for Facial Expression Recognition in Videos

paper_content:

We present a new action recognition deep neural network which adaptively learns the best action velocities in addition to the classification. While deep neural networks have reached maturity for image understanding tasks, we are still exploring network topologies and features to handle the richer environment of video clips. Here, we tackle the problem of multiple velocities in action recognition, and provide state-of-the-art results for facial expression recognition, on known and new collected datasets. We further provide the training steps for our semi-supervised network, suited to learn from huge unlabeled datasets with only a fraction of labeled examples.
---
paper_title: Illumination invariants in deep video expression recognition

paper_content:

Abstract In this paper we present architectures based on deep neural nets for expression recognition in videos, which are invariant to local scaling. We amalgamate autoencoder and predictor architectures using an adaptive weighting scheme coping with a reduced size labeled dataset, while enriching our models from enormous unlabeled sets. We further improve robustness to lighting conditions by introducing a new adaptive filter based on temporal local scale normalization. We provide superior results over known methods, including recent reported approaches based on neural nets.
---
paper_title: Illumination invariants in deep video expression recognition

paper_content:

Abstract In this paper we present architectures based on deep neural nets for expression recognition in videos, which are invariant to local scaling. We amalgamate autoencoder and predictor architectures using an adaptive weighting scheme coping with a reduced size labeled dataset, while enriching our models from enormous unlabeled sets. We further improve robustness to lighting conditions by introducing a new adaptive filter based on temporal local scale normalization. We provide superior results over known methods, including recent reported approaches based on neural nets.
---
paper_title: Web-based database for facial expression analysis

paper_content:

In the last decade, the research topic of automatic analysis of facial expressions has become a central topic in machine vision research. Nonetheless, there is a glaring lack of a comprehensive, readily accessible reference set of face images that could be used as a basis for benchmarks for efforts in the field. This lack of easily accessible, suitable, common testing resource forms the major impediment to comparing and extending the issues concerned with automatic facial expression analysis. In this paper, we discuss a number of issues that make the problem of creating a benchmark facial expression database difficult. We then present the MMI facial expression database, which includes more than 1500 samples of both static images and image sequences of faces in frontal and in profile view displaying various expressions of emotion, single and multiple facial muscle activation. It has been built as a Web-based direct-manipulation application, allowing easy access and easy search of the available images. This database represents the most comprehensive reference set of images for studies on facial expression analysis to date.
---
paper_title: Affectiva-MIT Facial Expression Dataset (AM-FED): Naturalistic and Spontaneous Facial Expressions Collected "In-the-Wild"

paper_content:

Computer classification of facial expressions requires large amounts of data and this data needs to reflect the diversity of conditions seen in real applications. Public datasets help accelerate the progress of research by providing researchers with a benchmark resource. We present a comprehensively labeled dataset of ecologically valid spontaneous facial responses recorded in natural settings over the Internet. To collect the data, online viewers watched one of three intentionally amusing Super Bowl commercials and were simultaneously filmed using their webcam. They answered three self-report questions about their experience. A subset of viewers additionally gave consent for their data to be shared publicly with other researchers. This subset consists of 242 facial videos (168, 359 frames) recorded in real world conditions. The dataset is comprehensively labeled for the following: 1) frame-by-frame labels for the presence of 10 symmetrical FACS action units, 4 asymmetric (unilateral) FACS action units, 2 head movements, smile, general expressiveness, feature tracker fails and gender, 2) the location of 22 automatically detected landmark points, 3) self-report responses of familiarity with, liking of, and desire to watch again for the stimuli videos and 4) baseline performance of detection algorithms on this dataset. This data is available for distribution to researchers online, the EULA can be found at: http://www.affectiva.com/facial-expression-dataset-am-fed/.
---
paper_title: The CAS-PEAL Large-Scale Chinese Face Database and Baseline Evaluations

paper_content:

In this paper, we describe the acquisition and contents of a large-scale Chinese face database: the CAS-PEAL face database. The goals of creating the CAS-PEAL face database include the following: 1) providing the worldwide researchers of face recognition with different sources of variations, particularly pose, expression, accessories, and lighting (PEAL), and exhaustive ground-truth information in one uniform database; 2) advancing the state-of-the-art face recognition technologies aiming at practical applications by using off-the-shelf imaging equipment and by designing normal face variations in the database; and 3) providing a large-scale face database of Mongolian. Currently, the CAS-PEAL face database contains 99 594 images of 1040 individuals (595 males and 445 females). A total of nine cameras are mounted horizontally on an arc arm to simultaneously capture images across different poses. Each subject is asked to look straight ahead, up, and down to obtain 27 images in three shots. Five facial expressions, six accessories, and 15 lighting changes are also included in the database. A selected subset of the database (CAS-PEAL-R1, containing 30 863 images of the 1040 subjects) is available to other researchers now. We discuss the evaluation protocol based on the CAS-PEAL-R1 database and present the performance of four algorithms as a baseline to do the following: 1) elementarily assess the difficulty of the database for face recognition algorithms; 2) preference evaluation results for researchers using the database; and 3) identify the strengths and weaknesses of the commonly used algorithms.
---
paper_title: The Vera am Mittag German audio-visual emotional speech database

paper_content:

The lack of publicly available annotated databases is one of the major barriers to research advances on emotional information processing. In this contribution we present a recently collected database of spontaneous emotional speech in German which is being made available to the research community. The database consists of 12 hours of audio-visual recordings of the German TV talk show ldquoVera am Mittagrdquo, segmented into broadcasts, dialogue acts and utterances. This corpus contains spontaneous and very emotional speech recorded from unscripted, authentic discussions between the guests of the talk show. In addition to the audio-visual data and the segmented utterances we provide emotion labels for a great part of the data. The emotion labels are given on a continuous valued scale for three emotion primitives: valence, activation and dominance, using a large number of human evaluators. Such data is of great interest to all research groups working on spontaneous speech analysis, emotion recognition in both speech and facial expression, natural language understanding, and robust speech recognition.
---
paper_title: A Natural Visible and Infrared Facial Expression Database for Expression Recognition and Emotion Inference

paper_content:

To date, most facial expression analysis has been based on visible and posed expression databases. Visible images, however, are easily affected by illumination variations, while posed expressions differ in appearance and timing from natural ones. In this paper, we propose and establish a natural visible and infrared facial expression database, which contains both spontaneous and posed expressions of more than 100 subjects, recorded simultaneously by a visible and an infrared thermal camera, with illumination provided from three different directions. The posed database includes the apex expressional images with and without glasses. As an elementary assessment of the usability of our spontaneous database for expression recognition and emotion inference, we conduct visible facial expression recognition using four typical methods, including the eigenface approach [principle component analysis (PCA)], the fisherface approach [PCA + linear discriminant analysis (LDA)], the Active Appearance Model (AAM), and the AAM-based + LDA. We also use PCA and PCA+LDA to recognize expressions from infrared thermal images. In addition, we analyze the relationship between facial temperature and emotion through statistical analysis. Our database is available for research purposes.
---
paper_title: Differentiating spontaneous from posed facial expressions within a generic facial expression recognition framework

paper_content:

In this paper we propose the first method known to the authors that successfully differentiates spontaneous from posed facial expressions using a realistic training corpus. We propose a new spatiotemporal local texture descriptor (CLBP-TOP) that outperforms other descriptors. We demonstrate that our temporal interpolation and visual/near-infrared fusion methods improve the differentiation performance. Finally, we propose a new generic facial expression recognition framework that subdivides the facial expression recognition problem into a cascade of smaller tasks that are simpler to tackle. The system is the first to differentiate spontaneous from posed facial expressions with a realistic corpus and achieves promising results.
---
paper_title: Multimodal Spontaneous Emotion Corpus for Human Behavior Analysis

paper_content:

Emotion is expressed in multiple modalities, yet most research has considered at most one or two. This stems in part from the lack of large, diverse, well-annotated, multimodal databases with which to develop and test algorithms. We present a well-annotated, multimodal, multidimensional spontaneous emotion corpus of 140 participants. Emotion inductions were highly varied. Data were acquired from a variety of sensors of the face that included high-resolution 3D dynamic imaging, high-resolution 2D video, and thermal (infrared) sensing, and contact physiological sensors that included electrical conductivity of the skin, respiration, blood pressure, and heart rate. Facial expression was annotated for both the occurrence and intensity of facial action units from 2D video by experts in the Facial Action Coding System (FACS). The corpus further includes derived features from 3D, 2D, and IR (infrared) sensors and baseline results for facial expression and action unit detection. The entire corpus will be made available to the research community.
---
paper_title: Learning Expressionlets on Spatio-temporal Manifold for Dynamic Facial Expression Recognition

paper_content:

Facial expression is temporally dynamic event which can be decomposed into a set of muscle motions occurring in different facial regions over various time intervals. For dynamic expression recognition, two key issues, temporal alignment and semantics-aware dynamic representation, must be taken into account. In this paper, we attempt to solve both problems via manifold modeling of videos based on a novel mid-level representation, i.e. expressionlet. Specifically, our method contains three key components: 1) each expression video clip is modeled as a spatio-temporal manifold (STM) formed by dense low-level features, 2) a Universal Manifold Model (UMM) is learned over all low-level features and represented as a set of local ST modes to statistically unify all the STMs. 3) the local modes on each STM can be instantiated by fitting to UMM, and the corresponding expressionlet is constructed by modeling the variations in each local ST mode. With above strategy, expression videos are naturally aligned both spatially and temporally. To enhance the discriminative power, the expressionlet-based STM representation is further processed with discriminant embedding. Our method is evaluated on four public expression databases, CK+, MMI, Oulu-CASIA, and AFEW. In all cases, our method reports results better than the known state-of-the-art.
---
paper_title: Why is facial expression analysis in the wild challenging?

paper_content:

In this paper, we discuss the challenges for facial expression analysis in the wild. We studied the problems exemplarily on the Emotion Recognition in the Wild Challenge 2013 [3] dataset. We performed extensive experiments on this dataset comparing different approaches for face alignment, face representation, and classification, as well as human performance. It turns out that under close-to-real conditions, especially with co-occurring speech, it is hard even for humans to assign emotion labels to clips when only taking video into account. Our experiments on automatic emotion classification achieved at best a correct classification rate of 29.81% on the test set using Gabor features and linear support vector machines, which were trained on web images. This result is 7.06% better than the official baseline, which additionally incorporates time information.
---
paper_title: Real-time facial expression recognition based on pixel-pattern-based texture feature

paper_content:

Proposed is a pixel-pattern-based texture feature (PPBTF) for realtime facial expression recognition. Grey-scale images are transformed into pattern maps where edges and lines are used for characterising facial texture. Based on the pattern map, a feature vector is constructed. Adaboost and a support vector machine (SVM) are adopted. Experiments on the Cohn-Kanade database illustrate that the PPBTF is effective and efficient for facial expression recognition.
---
paper_title: Facial Expression Recognition From Image Sequence Based on LBP and Taylor Expansion

paper_content:

The aim of an automatic video-based facial expression recognition system is to detect and classify human facial expressions from image sequence. An integrated automatic system often involves two components: 1) peak expression frame detection and 2) expression feature extraction. In comparison with the image-based expression recognition system, the video-based recognition system often performs online detection, which prefers low-dimensional feature representation for cost-effectiveness. Moreover, effective feature extraction is needed for classification. Many recent recognition systems often incorporate rich additional subjective information and thus become less efficient for real-time application. In our facial expression recognition system, first, we propose the double local binary pattern (DLBP) to detect the peak expression frame from the video. The proposed DLBP method has a much lower-dimensional size and can successfully reduce detection time. Besides, to handle the illumination variations in LBP, logarithm-laplace (LL) domain is further proposed to get a more robust facial feature for detection. Finally, the Taylor expansion theorem is employed in our system for the first time to extract facial expression feature. We propose the Taylor feature pattern (TFP) based on the LBP and Taylor expansion to obtain an effective facial feature from the Taylor feature map. Experimental results on the JAFFE and Cohn-Kanade data sets show that the proposed TFP method outperforms some state-of-the-art LBP-based feature extraction methods for facial expression feature extraction and can be suited for real-time applications.
---
paper_title: Facial Expression Recognition Utilizing Local Direction-Based Robust Features and Deep Belief Network

paper_content:

Emotional health plays very vital role to improve people’s quality of lives, especially for the elderly. Negative emotional states can lead to social or mental health problems. To cope with emotional health problems caused by negative emotions in daily life, we propose efficient facial expression recognition system to contribute in emotional healthcare system. Thus, facial expressions play a key role in our daily communications, and recent years have witnessed a great amount of research works for reliable facial expressions recognition (FER) systems. Therefore, facial expression evaluation or analysis from video information is very challenging and its accuracy depends on the extraction of robust features. In this paper, a unique feature extraction method is presented to extract distinguished features from the human face. For person independent expression recognition, depth video data is used as input to the system where in each frame, pixel intensities are distributed based on the distances to the camera. A novel robust feature extraction process is applied in this work which is named as local directional position pattern (LDPP). In LDPP, after extracting local directional strengths for each pixel such as applied in typical local directional pattern (LDP), top directional strength positions are considered in binary along with their strength sign bits. Considering top directional strength positions with strength signs in LDPP can differentiate edge pixels with bright as well as dark regions on their opposite sides by generating different patterns whereas typical LDP only considers directions representing the top strengths irrespective of their signs as well as position orders (i.e., directions with top strengths represent 1 and rest of them 0), which can generate the same patterns in this regard sometimes. Hence, LDP fails to distinguish edge pixels with opposite bright and dark regions in some cases which can be overcome by LDPP. Moreover, the LDPP capabilities are extended through principal component analysis (PCA) and generalized discriminant analysis (GDA) for better face characteristic illustration in expression. The proposed features are finally applied with deep belief network (DBN) for expression training and recognition.
---
paper_title: Collaborative discriminative multi-metric learning for facial expression recognition in video

paper_content:

Abstract Facial expression recognition in video has been an important and relatively new topic in human face analysis and attracted growing interests in recent years. Unlike conventional image-based facial expression recognition methods which recognize facial expression category from still images, facial expression recognition in video is more challenging because there are usually larger intra-class variations among facial frames within a video. This paper presents a collaborative discriminative multi-metric learning (CDMML) for facial expression recognition in video. We first compute multiple feature descriptors for each face video to describe facial appearance and motion information from different aspects. Then, we learn multiple distance metrics with these extracted multiple features collaboratively to exploit complementary and discriminative information for recognition. Experimental results on the Acted Facial Expression in Wild (AFEW) 4.0 and the extended Cohn–Kanada (CK+) datasets are presented to demonstrate the effectiveness of our proposed method.
---
paper_title: Facial Expression Recognition Based on Deep Evolutional Spatial-Temporal Networks

paper_content:

One key challenging issue of facial expression recognition is to capture the dynamic variation of facial physical structure from videos. In this paper, we propose a part-based hierarchical bidirectional recurrent neural network (PHRNN) to analyze the facial expression information of temporal sequences. Our PHRNN models facial morphological variations and dynamical evolution of expressions, which is effective to extract “temporal features” based on facial landmarks (geometry information) from consecutive frames. Meanwhile, in order to complement the still appearance information, a multi-signal convolutional neural network (MSCNN) is proposed to extract “spatial features” from still frames. We use both recognition and verification signals as supervision to calculate different loss functions, which are helpful to increase the variations of different expressions and reduce the differences among identical expressions. This deep evolutional spatial-temporal network (composed of PHRNN and MSCNN) extracts the partial-whole, geometry-appearance, and dynamic-still information, effectively boosting the performance of facial expression recognition. Experimental results show that this method largely outperforms the state-of-the-art ones. On three widely used facial expression databases (CK+, Oulu-CASIA, and MMI), our method reduces the error rates of the previous best ones by 45.5%, 25.8%, and 24.4%, respectively.
---
paper_title: Illumination Invariant Facial Expression Recognition Using Selected Merged Binary Patterns for Real World Images

paper_content:

Abstract Automatic facial expression recognition has always been a challenging task to understand human behavior from real world images. Certain type of issues are associated with such images that include poor illumination, different orientations and varying pose. The proposed technique first applies Fast Fourier Transform and Contrast Limited Adaptive Histogram Equalization (FFT + CLAHE) method to compensate the poor illumination. Then merged binary pattern code (MBPC) is generated for every pixel. Two bits per neighbourhood are produced to form a 16-bit code per pixel. This code merges local features to improve the effectiveness of facial expression recognition system. MBPC descriptor captures changes along fine edges and prominent pattern around eyes, eye brows, mouth, bulges and wrinkles of the face. The results of proposed technique are compared with different variants of LBP and LGC based techniques for both holistic and zoned images. Static Facial Expression in Wild (SFEW) dataset is selected for experimentation. Results clearly indicate that the suggested MBPC based technique surpasses other techniques with 96.5% and 67.2% accuracy for holistic and division based approach respectively. Moreover, results indicate that the performance of holistic approach is much higher than division based approach.
---
paper_title: Feature Fusion Based Audio-Visual Speaker Identification Using Hidden Markov Model under Different Lighting Variations

paper_content:

The aim of the paper is to propose a feature fusion based Audio-Visual Speaker Identification (AVSI) system with varied conditions of illumination environments. Among the different fusion strategies, feature level fusion has been used for the proposed AVSI system where Hidden Markov Model (HMM) is used for learning and classification. Since the feature set contains richer information about the raw biometric data than any other levels, integration at feature level is expected to provide better authentication results. In this paper, both Mel Frequency Cepstral Coefficients (MFCCs) and Linear Prediction Cepstral Coefficients (LPCCs) are combined to get the audio feature vectors and Active Shape Model (ASM) based appearance and shape facial features are concatenated to take the visual feature vectors. These combined audio and visual features are used for the feature-fusion. To reduce the dimension of the audio and visual feature vectors, Principal Component Analysis (PCA) method is used. The VALID audio-visual database is used to measure the performance of the proposed system where four different illumination levels of lighting conditions are considered. Experimental results focus on the significance of the proposed audio-visual speaker identification system with various combinations of audio and visual features.
---
paper_title: The more the merrier: Analysing the affect of a group of people in images

paper_content:

The recent advancement of social media has given users a platform to socially engage and interact with a global population. With millions of images being uploaded onto social media platforms, there is an increasing interest in inferring the emotion and mood display of a group of people in images. Automatic affect analysis research has come a long way but has traditionally focussed on a single subject in a scene. In this paper, we study the problem of inferring the emotion of a group of people in an image. This group affect has wide applications in retrieval, advertisement, content recommendation and security. The contributions of the paper are: 1) a novel emotion labelled database of groups of people in images; 2) a Multiple Kernel Learning based hybrid affect inference model; 3) a scene context based affect inference model; 4) a user survey to better understand the attributes that affect the perception of affect of a group of people in an image. The detailed experimentation validation provides a rich baseline for the proposed database.
---
paper_title: Spontaneous Facial Expression Recognition by Fusing Thermal Infrared and Visible Images

paper_content:

In this paper, we propose a method for spontaneous facial expression recognition by fusing features extracted from visible and thermal infrared images. First, the active appearance model parameters and head motion features are extracted from the visible images, and several thermal statistical parameters are extracted from the infrared images. Second, a multiple genetic algorithms-based fusion method is proposed for fusing these two spectrums. We use this proposed fusion method to search for the optimal combination of a similarity measurement and a feature subset. Then, a k-nearest neighbors classifier with the optimal combination is used to classify spontaneous facial expressions. Comparative experiments implemented on the Natural Visible and Infrared Facial Expression database show the effectiveness of the proposed similarity measurement and the feature selection method, and demonstrate the fusion method’s advantage over only using visible features.
---
paper_title: Illumination invariants in deep video expression recognition

paper_content:

Abstract In this paper we present architectures based on deep neural nets for expression recognition in videos, which are invariant to local scaling. We amalgamate autoencoder and predictor architectures using an adaptive weighting scheme coping with a reduced size labeled dataset, while enriching our models from enormous unlabeled sets. We further improve robustness to lighting conditions by introducing a new adaptive filter based on temporal local scale normalization. We provide superior results over known methods, including recent reported approaches based on neural nets.
---
paper_title: AffectiveWear: toward recognizing facial expression

paper_content:

Facial expression is a powerful way for us to exchange information nonverbally. They can give us insights into how people feel and think. There are a number of works related to facial expression detection in computer vision. However, most works focus on camera-based systems installed in the environment. With this method, it is difficult to track user's face if user moves constantly. Moreover, user's facial expression can be recognized at only a limited place.
---
paper_title: A facial expression recognition system using robust face features from depth videos and deep learning

paper_content:

Abstract This work proposes a depth camera-based robust facial expression recognition (FER) system that can be adopted for better human machine interaction. Although video-based facial expression analysis has been focused on by many researchers, there are still various problems to be solved in this regard such as noise due to illumination variations over time. Depth video data in the helps to make an FER system person-independent as pixel values in depth images are distributed based on distances from a depth camera. Besides, depth images should resolve some privacy issues as real identity of a user can be hidden. The accuracy of an FER system is much dependent on the extraction of robust features. Here, we propose a novel method to extract salient features from depth faces that are further combined with deep learning for efficient training and recognition. Eight directional strengths are obtained for each pixel in a depth image where signs of some top strengths are arranged to represent unique as well as robust face features, which can be denoted as Modified Local Directional Patterns (MLDP). The MLDP features are further processed by Generalized Discriminant Analysis (GDA) for better face feature extraction. GDA is an efficient tool that helps distinguishing MLDP features of different facial expressions by clustering the features from the same expression as close as possible and separating the features from different expressions as much as possible in a non-linear space. Then, MLDP-GDA features are applied with Deep Belief Network (DBN) for training different facial expressions. Finally, the trained DBN is used to recognize facial expressions in a depth video for testing. The proposed approach was compared with other traditional approaches in a standalone system where the proposed one showed its superiority by achieving mean recognition rate of 96.25% where the other approaches could make 91.67% at the best. The deep learning-based training and recognition of the facial expression features can also be undertaken with cloud computing to support many users and make the system faster than a standalone system.
---
paper_title: Facial Expression Recognition in Video with Multiple Feature Fusion

paper_content:

Video based facial expression recognition has been a long standing problem and attracted growing attention recently. The key to a successful facial expression recognition system is to exploit the potentials of audiovisual modalities and design robust features to effectively characterize the facial appearance and configuration changes caused by facial motions. We propose an effective framework to address this issue in this paper. In our study, both visual modalities (face images) and audio modalities (speech) are utilized. A new feature descriptor called Histogram of Oriented Gradients from Three Orthogonal Planes (HOG-TOP) is proposed to extract dynamic textures from video sequences to characterize facial appearance changes. And a new effective geometric feature derived from the warp transformation of facial landmarks is proposed to capture facial configuration changes. Moreover, the role of audio modalities on recognition is also explored in our study. We applied the multiple feature fusion to tackle the video-based facial expression recognition problems under lab-controlled environment and in the wild, respectively. Experiments conducted on the extended Cohn-Kanade (CK+) database and the Acted Facial Expression in Wild (AFEW) 4.0 database show that our approach is robust in dealing with video-based facial expression recognition problems under lab-controlled environment and in the wild compared with the other state-of-the-art methods.
---
paper_title: Robust facial expression recognition using RGB-D images and multichannel features

paper_content:

Traditional and classical methods of facial expression recognition are mainly based on intensity image and are prone to be disturbed by illumination, poses, and disguise. This research aims to develop a robust facial expression recognition method using RGB-D images and multichannel features. Based on image entropy and visual saliency, facial texture features are firstly extracted from RGB images and depth images to construct the Histogram of Oriented Gradient (HOG) descriptors. And then, we extract geometric features of RGB images using Active Appearance Model (AAM). Combining the HOG texture features with the AAM geometric feature, we build a robust multichannel feature vector for facial expression recognition. On this basis, an improved Support Vector Machine (SVM) algorithm, namely GS-SVM, is used to classify facial expression recognition. The proposed GS-SVM algorithm applies Grid Search method to optimize the best parameters for SVM classifier and estimate the accuracy of each parameter combination in specified range. Finally, the proposed methods are tested and evaluated on the merged RGB-D database. Experimental results show that the proposed algorithm not only achieves a higher average recognition rate but also is robust to uncontrolled environments.
---
paper_title: Facial Expression Recognition Utilizing Local Direction-Based Robust Features and Deep Belief Network

paper_content:

Emotional health plays very vital role to improve people’s quality of lives, especially for the elderly. Negative emotional states can lead to social or mental health problems. To cope with emotional health problems caused by negative emotions in daily life, we propose efficient facial expression recognition system to contribute in emotional healthcare system. Thus, facial expressions play a key role in our daily communications, and recent years have witnessed a great amount of research works for reliable facial expressions recognition (FER) systems. Therefore, facial expression evaluation or analysis from video information is very challenging and its accuracy depends on the extraction of robust features. In this paper, a unique feature extraction method is presented to extract distinguished features from the human face. For person independent expression recognition, depth video data is used as input to the system where in each frame, pixel intensities are distributed based on the distances to the camera. A novel robust feature extraction process is applied in this work which is named as local directional position pattern (LDPP). In LDPP, after extracting local directional strengths for each pixel such as applied in typical local directional pattern (LDP), top directional strength positions are considered in binary along with their strength sign bits. Considering top directional strength positions with strength signs in LDPP can differentiate edge pixels with bright as well as dark regions on their opposite sides by generating different patterns whereas typical LDP only considers directions representing the top strengths irrespective of their signs as well as position orders (i.e., directions with top strengths represent 1 and rest of them 0), which can generate the same patterns in this regard sometimes. Hence, LDP fails to distinguish edge pixels with opposite bright and dark regions in some cases which can be overcome by LDPP. Moreover, the LDPP capabilities are extended through principal component analysis (PCA) and generalized discriminant analysis (GDA) for better face characteristic illustration in expression. The proposed features are finally applied with deep belief network (DBN) for expression training and recognition.
---
paper_title: Collaborative discriminative multi-metric learning for facial expression recognition in video

paper_content:

Abstract Facial expression recognition in video has been an important and relatively new topic in human face analysis and attracted growing interests in recent years. Unlike conventional image-based facial expression recognition methods which recognize facial expression category from still images, facial expression recognition in video is more challenging because there are usually larger intra-class variations among facial frames within a video. This paper presents a collaborative discriminative multi-metric learning (CDMML) for facial expression recognition in video. We first compute multiple feature descriptors for each face video to describe facial appearance and motion information from different aspects. Then, we learn multiple distance metrics with these extracted multiple features collaboratively to exploit complementary and discriminative information for recognition. Experimental results on the Acted Facial Expression in Wild (AFEW) 4.0 and the extended Cohn–Kanada (CK+) datasets are presented to demonstrate the effectiveness of our proposed method.
---
paper_title: An IR-based facial expression tracking sensor for head-mounted displays

paper_content:

We proposed an IR-based facial expression tracking sensor for Head-Mounted Displays(HMD). The proposed sensor uses lateral propagation characteristics of IR light on human skin to capture the degree of compressed or stretched deformations of facial skin. We derived a semi-empirical equation modeling the lateral propagation characteristics of vertically incident IR light into human skin, with parameters which fit to the measured data. All components of IR emitters and receivers were integrated into the form interface of a commercial VR headset. We verified the functionality of tracking performance from 4-kind of basic facial expressions in the experiment using the implemented prototype headset. As a further work, we will upgrade the performance of proposed sensor to recognize emotional expressions of HMD user by applying machine learning technique. We are aiming to enable immersive human to human(or avatar) communications in the cyberspace.
---
paper_title: Facial Expression Recognition from Infrared Thermal Videos

paper_content:

In this paper, a spontaneous facial expression recognition method using infrared thermal videos is proposed. Firstly, the sequence features are extracted from the infrared thermal horizontal and vertical temperature difference sequences of different facial sub-regions. Secondly, a feature subset is selected according to their F-values. Thirdly, the Adaboost algorithm, with the weak classifiers of k-Nearest Neighbor, is used to classify facial expressions in arousal and valence dimensions. Finally, experiments on the Natural Visible and Infrared facial Expression (USTC-NVIE) database demonstrates the effectiveness of our approach.
---
paper_title: Spontaneous Facial Expression Recognition by Fusing Thermal Infrared and Visible Images

paper_content:

In this paper, we propose a method for spontaneous facial expression recognition by fusing features extracted from visible and thermal infrared images. First, the active appearance model parameters and head motion features are extracted from the visible images, and several thermal statistical parameters are extracted from the infrared images. Second, a multiple genetic algorithms-based fusion method is proposed for fusing these two spectrums. We use this proposed fusion method to search for the optimal combination of a similarity measurement and a feature subset. Then, a k-nearest neighbors classifier with the optimal combination is used to classify spontaneous facial expressions. Comparative experiments implemented on the Natural Visible and Infrared Facial Expression database show the effectiveness of the proposed similarity measurement and the feature selection method, and demonstrate the fusion method’s advantage over only using visible features.
---
paper_title: The Emotracker: Visualizing Contents, Gaze and Emotions at a Glance

paper_content:

In the last years, Affective Computing investigations have focused the efforts in the automatic extraction of human emotions and in increasing the success rates in the emotion recognition task. However, there is a lack of automatic tools that intuitively visualize the users' emotional information. In this work, we propose Emotracker, a novel tool based on the combination of eye tracking and facial emotional recognition technologies that allows the visualization of contents, user emotions and gaze at a glance, and offers a wide range of visualization options, including emotional saccade maps and emotional heat maps. The simultaneous consideration of gaze and emotion opens the door to the evaluation of user engagement, which is of great importance for content creators, but has also other promising fields of application in psychology research and, more specifically, in autism.
---
paper_title: Posed and spontaneous expression recognition through modeling their spatial patterns

paper_content:

This paper presents a new method to recognize posed and spontaneous expressions through modeling their spatial patterns. Gender and expression categories are employed as privileged information to further improve the recognition. The proposed approach includes three steps. First, geometric features about facial shape and Action Unit variations are extracted from the differences between apex and onset facial images to capture the spatial facial variation. Second, statistical hypothesis testings are conducted to explore the differences between posed and spontaneous expressions using the defined geometric features from three aspects: all samples, samples given the gender information, and samples given expression categories. Third, several Bayesian networks are built to capture posed and spontaneous spatial facial patterns respectively given gender and expression categories. The statistical analysis results on the USTC-NVIE and SPOS databases both demonstrate the effectiveness of the proposed geometric features. The recognition results on the USTC-NVIE database indicate that the privileged information of gender and expression can help model the spatial patterns caused by posed and spontaneous expressions. The recognition results on both databases outperform those of the state of the art.
---
paper_title: A Polarimetric Thermal Database for Face Recognition Research

paper_content:

We present a polarimetric thermal face database, the first of its kind, for face recognition research. This database was acquired using a polarimetric longwave infrared imager, specifically a division-of-time spinning achromatic retarder system. A corresponding set of visible spectrum imagery was also collected, to facilitate crossspectrum (also referred to as heterogeneous) face recognition research. The database consists of imagery acquired at three distances under two experimental conditions: neutral/baseline condition, and expressions condition. Annotations (spatial coordinates of key fiducial points) are provided for all images. Cross-spectrum face recognition performance on the database is benchmarked using three techniques: partial least squares, deep perceptual mapping, and coupled neural networks.
---

Title: A Review on Automatic Facial Expression Recognition Systems Assisted by Multimodal Sensor Data

Section 1: Introduction
Description 1: This section introduces the importance of facial expression recognition systems and their applications in various fields, such as human-computer interaction, healthcare, and human-robot interaction.

Section 2: Motivation
Description 2: This section discusses the broad applications of FER systems and the challenges faced by these systems in real-world environments, including illumination variation, subject-dependence, and head pose changes.

Section 3: Existing FER Systems
Description 3: This section reviews the existing FER systems, their advantages, and disadvantages, and categorizes them based on their evaluation in lab-controlled or real-world environments.

Section 4: Illumination Variation
Description 4: This section explains how illumination variation affects FER systems and reviews state-of-the-art methods used to tackle this challenge.

Section 5: Subject-Dependence
Description 5: This section describes the challenge of subject-dependence in FER systems and reviews existing methods to address this issue.

Section 6: Head Pose
Description 6: This section discusses the challenge of head pose variations in FER systems and reviews various approaches to handle this problem.

Section 7: Multimodal Sensors
Description 7: This section categorizes sensors used in FER systems into three classes: detailed-face, non-visual, and target-focused sensors, and explains how these sensors can improve FER performance.

Section 8: Datasets
Description 8: This section describes various datasets used in FER research, categorizing them according to the types of sensors used and discussing their suitability for real-world applications.

Section 9: Discussion and Comparison of Existing Real-World, Multimodal FER Systems
Description 9: This section compares and discusses the existing FER systems that use multimodal sensors, highlighting their strengths and weaknesses and their effectiveness in real-world environments.

Section 10: The Proposed Method and Future Direction
Description 10: This section proposes a multimodal sensor framework to improve FER systems' accuracy and robustness in real-world environments and discusses future directions for research.

Section 11: Open Problems
Description 11: This section identifies open problems in the FER field, such as subject-dependence, large pose variations, and multimodal sensor data fusion, and suggests areas for future investigation.

Section 12: Conclusions and Future Work
Description 12: This section summarizes the findings of the review, discusses the potential of the proposed multimodal sensor framework, and suggests future work to improve FER systems.