A Survey on Human Emotion Recognition Approaches, Databases and Applications
---
paper_title: A survey of affect recognition methods: audio, visual and spontaneous expressions

paper_content:

Automated analysis of human affective behavior has attracted increasing attention from researchers in psychology, computer science, linguistics, neuroscience, and related disciplines. Promising approaches have been reported, including automatic methods for facial and vocal affect recognition. However, the existing methods typically handle only deliberately displayed and exaggerated expressions of prototypical emotions-despite the fact that deliberate behavior differs in visual and audio expressions from spontaneously occurring behavior. Recently efforts to develop algorithms that can process naturally occurring human affective behavior have emerged. This paper surveys these efforts. We first discuss human emotion perception from a psychological perspective. Next, we examine the available approaches to solving the problem of machine understanding of human affective behavior occurring in real-world settings. We finally outline some scientific and engineering challenges for advancing human affect sensing technology.
---
paper_title: Affect Detection: An Interdisciplinary Review of Models, Methods, and Their Applications

paper_content:

This survey describes recent progress in the field of Affective Computing (AC), with a focus on affect detection. Although many AC researchers have traditionally attempted to remain agnostic to the different emotion theories proposed by psychologists, the affective technologies being developed are rife with theoretical assumptions that impact their effectiveness. Hence, an informed and integrated examination of emotion theories from multiple areas will need to become part of computing practice if truly effective real-world systems are to be achieved. This survey discusses theoretical perspectives that view emotions as expressions, embodiments, outcomes of cognitive appraisal, social constructs, products of neural circuitry, and psychological interpretations of basic feelings. It provides meta-analyses on existing reviews of affect detection systems that focus on traditional affect detection modalities like physiology, face, and voice, and also reviews emerging research on more novel channels such as text, body language, and complex multimodal systems. This survey explicitly explores the multidisciplinary foundation that underlies all AC applications by describing how AC researchers have incorporated psychological theories of emotion and how these theories affect research questions, methods, results, and their interpretations. In this way, models and methods can be compared, and emerging insights from various disciplines can be more expertly integrated.
---
paper_title: Affective computing: Challenges

paper_content:

A number of researchers around the world have built machines that recognize, express, model, communicate, and respond to emotional information, instances of "affective computing." This article raises and responds to several criticisms of affective computing, articulating state-of-the art research challenges, especially with respect to affect in human-computer interaction.
---
paper_title: Handbook of Emotions

paper_content:

Part 1. Interdisciplinary Foundations. R.C. Solomon, The Philosophy of Emotions. P.N. Stearns, History of Emotions: Issues of Change and Impact. J.E. Stets, J.H. Turner, The Sociology of Emotions. J. Panksepp, The Affective Brain and Core Consciousness: How Does Neural Activity Generate Emotional Feelings? N.H. Frijda, The Psychologist's Point of View. L.S. Greenberg, The Clinical Application of Emotion in Psychotherapy. P.N. Johnson-Laird, K. Oatley, Emotions, Music, and Literature. J. Tooby, L. Cosmides, The Evolutionary Psychology of the Emotions and Their Relationship to Internal Regulatory Variables. R. Loewenstein, G. Loewenstein, The Role of Emotion in Economic Behavior. Part 2. Biological and Neurophysiological Approaches to Emotion. J.E. LeDoux, E.A. Phelps, Emotional Networks in the Brain. J.T. Larsen, G.G. Berntson, K.M. Poehlmann, T.A. Ito, J.T. Cacioppo, The Psychophysiology of Emotion. J. Bachorowski, M.J. Owren, Vocal Expressions of Emotion. D. Matsumoto, D. Keltner, M.N. Shiota, M. O'Sullivan, M. Frank, Facial Expressions of Emotion. J.M. Haviland-Jones, P.J. Wilson, A "Nose" for Emotion: Emotional Information and Challenges in Odors and Semiochemicals. T.D. Wager, L. Feldman Barrett, E. Bliss-Moreau, K. Lindquist, S. Duncan, H. Kober, J. Joseph, M. Davidson, J. Mize, The Neuroimaging of Emotion. A.D. Craig, Interoception and Emotion: A Neuroanatomical Perspective. Part 3. Developmental Changes. L.A. Camras, S.S. Fatani, The Development of Facial Expressions: Current Perspectives on Infant Emotions. M. Lewis, The Emergence of Human Emotions. P.L. Harris, Children's Understanding of Emotion. C. Saarni, The Interface of Emotional Development with Social Context. S.C. Widen, J.A. Russell, Young Children's Understanding of Others' Emotions. A.S. Walker-Andrews, Intermodal Emotional Processes in Infancy. C. Magai, Long-Lived Emotions: A Lifecourse Perspective on Emotional Development. Part 4. Social Perspectives. L.R. Brody, J.A. Hall, Gender and Emotion in Context. R.A. Shweder, J. Haidt, R. Horton, C. Joseph, The Cultural Psychology of the Emotions: Ancient and Renewed. E.R. Smith, D.M. Mackie, Intergroup Emotions. M.L. Hoffman, Empathy and Prosocial Behavior. A.H. Fischer, A.S.R. Manstead, Social Functions of Emotion. Part 5. Personality Issues. R.E. Lucas, E. Diener, Subjective Well-Being. J.E. Bates, J.A. Goodnight, J.E. Fite, Temperament and Emotion. J.J. Gross, Emotion Regulation. K.A. Lindquist, L. Feldman Barrett, Emotional Complexity. Part 6. Cognitive Factors. P. Salovey, B.T. Detweiler-Bedell, J.B. Detweiler-Bedell, J.D. Mayer, Emotional Intelligence. A.M. Isen, Some Ways in which Positive Affect Influences Decision Making and Problem Solving. N.L. Stein, M.W. Hernandez, T. Trabasso, Advances in Modeling Emotion and Thought: The Importance of Development, On-Line and Multilevel Analyses. P.M. Niedenthal, Emotion Concepts. E.A. Kensinger, D.L. Schacter, Memory and Emotion. M. Minsky, A Framework for Representing Emotional States. G.L. Clore, A. Ortony, Appraisal Theories: How Cognition Shapes Affect into Emotion. Part 7. Health and Emotions. M.A. Diefenbach, S.M. Miller, M. Porter, E. Peters, M. Stefanek, H. Leventhal, Emotions and Health Behavior: A Self-Regulation Perspective. M.E. Kemeny, A. Shestyuk, Emotions, the Neuroendocrine and Immune Systems, and Health. N.S. Consedine, Emotions and Health. A.M. Kring, Emotion Disturbances as Transdiagnostic Processes in Psychopathology. Part 8. Select Emotions. A. Ohman, Fear and Anxiety: Overlaps and Dissociations. E.A. Lemerise, K.A. Dodge, The Development of Anger and Hostile Interactions. M. Lewis, Self-Conscious Emotions: Embarrassment, Pride, Shame, and Guilt. P. Rozin, J. Haidt, C.R. McCauley, Disgust. B.L. Fredrickson, M.A. Cohn, Positive Emotions. G.A. Bonanno, L. Goorin, K.G. Coifman, Sadness and Grief.
---
paper_title: An affective computing approach to physiological emotion specificity: Toward subject-independent and stimulus-independent classification of film-induced emotions

paper_content:

The hypothesis of physiological emotion specificity has been tested using pattern classification analysis (PCA). To address limitations of prior research using PCA, we studied effects of feature selection (sequential forward selection, sequential backward selection), classifier type (linear and quadratic discriminant analysis, neural networks, k-nearest neighbors method), and cross-validation method (subject- and stimulus-(in)dependence). Analyses were run on a data set of 34 participants watching two sets of three 10-min film clips (fearful, sad, neutral) while autonomic, respiratory, and facial muscle activity were assessed. Results demonstrate that the three states can be classified with high accuracy by most classifiers, with the sparsest model having only five features, even for the most difficult task of identifying the emotion of an unknown subject in an unknown situation (77.5%). Implications for choosing PCA parameters are discussed. Descriptors: Emotion, Pattern classification, Feature selection, Autonomic nervous system, Cardiovascular system, Respiration, Electrodermal system, Affective neuroscience, Affective computing There exists a long tradition of psychophysiological research on differential physiological responding among emotions, originally based on James’ (1884) peripheral perception theory and revived by basic emotion theory (e.g., Ekman, Levenson, & Friesen, 1983; see Friedman, 2010, for a review). These theories hold as a central tenet that basic human emotions have distinct physiological patterns. In an influential series of publications, Fridlund and colleagues (Fridlund & Izard, 1983; Fridlund, Schwartz, & Fowler, 1984) argued for the advantages of a formal patternclassification approach for the study of such physiological patterns of emotion. This approach recognizes the interactive and
---
paper_title: Dynamic Texture Recognition Using Local Binary Patterns with an Application to Facial Expressions

paper_content:

Dynamic texture (DT) is an extension of texture to the temporal domain. Description and recognition of DTs have attracted growing attention. In this paper, a novel approach for recognizing DTs is proposed and its simplifications and extensions to facial image analysis are also considered. First, the textures are modeled with volume local binary patterns (VLBP), which are an extension of the LBP operator widely used in ordinary texture analysis, combining motion and appearance. To make the approach computationally simple and easy to extend, only the co-occurrences of the local binary patterns on three orthogonal planes (LBP-TOP) are then considered. A block-based method is also proposed to deal with specific dynamic events such as facial expressions in which local information and its spatial locations should also be taken into account. In experiments with two DT databases, DynTex and Massachusetts Institute of Technology (MIT), both the VLBP and LBP-TOP clearly outperformed the earlier approaches. The proposed block-based method was evaluated with the Cohn-Kanade facial expression database with excellent results. The advantages of our approach include local processing, robustness to monotonic gray-scale changes, and simple computation
---
paper_title: Facial-component-based bag of words and PHOG descriptor for facial expression recognition

paper_content:

A novel framework of facial appearance and shape information extraction for facial expression recognition is proposed. For appearance extraction, a facial-component-based bag of words method is presented. We segment face images into 4 component regions, and sub-divide them into 4×4 sub-regions. Dense SIFT (Scale-Invariant Feature Transform) features are calculated over the sub-regions and vector quantized into 4×4 sets of codeword distributions. For shape extraction, PHOG (Pyramid Histogram of Orientated Gradient) descriptors are computed on the 4 facial component regions to obtain the spatial distribution of edges. Our framework provides holistic characteristics for the local texture and shape features by enhancing the structure-based spatial information, and makes the local descriptors be possible to be used in facial expression recognition for the first time. The recognition rate achieved by the fusion of appearance and shape features at decision level using the Cohn-Kanade database is 96.33%, which outperforms the state of the arts.
---
paper_title: Facial expression recognition using tracked facial actions: Classifier performance analysis

paper_content:

In this paper, we address the analysis and recognition of facial expressions in continuous videos. More precisely, we study classifiers performance that exploit head pose independent temporal facial action parameters. These are provided by an appearance-based 3D face tracker that simultaneously provides the 3D head pose and facial actions. The use of such tracker makes the recognition pose- and texture-independent. Two different schemes are studied. The first scheme adopts a dynamic time warping technique for recognizing expressions where training data are given by temporal signatures associated with different universal facial expressions. The second scheme models temporal signatures associated with facial actions with fixed length feature vectors (observations), and uses some machine learning algorithms in order to recognize the displayed expression. Experiments quantified the performance of different schemes. These were carried out on CMU video sequences and home-made video sequences. The results show that the use of dimension reduction techniques on the extracted time series can improve the classification performance. Moreover, these experiments show that the best recognition rate can be above 90%.
---
paper_title: An Automatic Framework for Textured 3D Video-Based Facial Expression Recognition

paper_content:

Most of the existing research on 3D facial expression recognition has been done using static 3D meshes. 3D videos of a face are believed to contain more information in terms of the facial dynamics which are very critical for expression recognition. This paper presents a fully automatic framework which exploits the dynamics of textured 3D videos for recognition of six discrete facial expressions. Local video-patches of variable lengths are extracted from numerous locations of the training videos and represented as points on the Grassmannian manifold. An efficient graph-based spectral clustering algorithm is used to separately cluster these points for every expression class. Using a valid Grassmannian kernel function, the resulting cluster centers are embedded into a Reproducing Kernel Hilbert Space (RKHS) where six binary SVM models are learnt. Given a query video, we extract video-patches from it, represent them as points on the manifold and match these points with the learnt SVM models followed by a voting based strategy to decide about the class of the query video. The proposed framework is also implemented in parallel on 2D videos and a score level fusion of 2D & 3D videos is performed for performance improvement of the system. The experimental results on BU4DFE data set show that the system achieves a very high classification accuracy for facial expression recognition from 3D videos.
---
paper_title: Facial Expression Recognition Using Facial Movement Features

paper_content:

Facial expression is an important channel for human communication and can be applied in many real applications. One critical step for facial expression recognition (FER) is to accurately extract emotional features. Current approaches on FER in static images have not fully considered and utilized the features of facial element and muscle movements, which represent static and dynamic, as well as geometric and appearance characteristics of facial expressions. This paper proposes an approach to solve this limitation using "salient” distance features, which are obtained by extracting patch-based 3D Gabor features, selecting the "salient” patches, and performing patch matching operations. The experimental results demonstrate high correct recognition rate (CRR), significant performance improvements due to the consideration of facial element and muscle movements, promising results under face registration errors, and fast processing time. Comparison with the state-of-the-art performance confirms that the proposed approach achieves the highest CRR on the JAFFE database and is among the top performers on the Cohn-Kanade (CK) database.
---
paper_title: Simultaneous Facial Action Tracking and Expression Recognition in the Presence of Head Motion

paper_content:

The recognition of facial gestures and expressions in image sequences is an important and challenging problem. Most of the existing methods adopt the following paradigm. First, facial actions/features are retrieved from the images, then the facial expression is recognized based on the retrieved temporal parameters. In contrast to this mainstream approach, this paper introduces a new approach allowing the simultaneous retrieval of facial actions and expression using a particle filter adopting multi-class dynamics that are conditioned on the expression. For each frame in the video sequence, our approach is split into two consecutive stages. In the first stage, the 3D head pose is retrieved using a deterministic registration technique based on Online Appearance Models. In the second stage, the facial actions as well as the facial expression are simultaneously retrieved using a stochastic framework based on second-order Markov chains. The proposed fast scheme is either as robust as, or more robust than existing ones in a number of respects. We describe extensive experiments and provide evaluations of performance to show the feasibility and robustness of the proposed approach.
---
paper_title: Robust Representation and Recognition of Facial Emotions Using Extreme Sparse Learning

paper_content:

Recognition of natural emotions from human faces is an interesting topic with a wide range of potential applications, such as human-computer interaction, automated tutoring systems, image and video retrieval, smart environments, and driver warning systems. Traditionally, facial emotion recognition systems have been evaluated on laboratory controlled data, which is not representative of the environment faced in real-world applications. To robustly recognize the facial emotions in real-world natural situations, this paper proposes an approach called extreme sparse learning, which has the ability to jointly learn a dictionary (set of basis) and a nonlinear classification model. The proposed approach combines the discriminative power of extreme learning machine with the reconstruction property of sparse representation to enable accurate classification when presented with noisy signals and imperfect data recorded in natural settings. In addition, this paper presents a new local spatio-temporal descriptor that is distinctive and pose-invariant. The proposed framework is able to achieve the state-of-the-art recognition accuracy on both acted and spontaneous facial emotion databases.
---
paper_title: Affective Body Expression Perception and Recognition: A Survey

paper_content:

Thanks to the decreasing cost of whole-body sensing technology and its increasing reliability, there is an increasing interest in, and understanding of, the role played by body expressions as a powerful affective communication channel. The aim of this survey is to review the literature on affective body expression perception and recognition. One issue is whether there are universal aspects to affect expression perception and recognition models or if they are affected by human factors such as culture. Next, we discuss the difference between form and movement information as studies have shown that they are governed by separate pathways in the brain. We also review psychological studies that have investigated bodily configurations to evaluate if specific features can be identified that contribute to the recognition of specific affective states. The survey then turns to automatic affect recognition systems using body expressions as at least one input modality. The survey ends by raising open questions on data collecting, labeling, modeling, and setting benchmarks for comparing automatic recognition systems.
---
paper_title: Real-time inference of mental states from facial expressions and upper body gestures

paper_content:

We present a real-time system for detecting facial action units and inferring emotional states from head and shoulder gestures and facial expressions. The dynamic system uses three levels of inference on progressively longer time scales. Firstly, facial action units and head orientation are identified from 22 feature points and Gabor filters. Secondly, Hidden Markov Models are used to classify sequences of actions into head and shoulder gestures. Finally, a multi level Dynamic Bayesian Network is used to model the unfolding emotional state based on probabilities of different gestures. The most probable state over a given video clip is chosen as the label for that clip. The average F1 score for 12 action units (AUs 1, 2, 4, 6, 7, 10, 12, 15, 17, 18, 25, 26), labelled on a frame by frame basis, was 0.461. The average classification rate for five emotional states (anger, fear, joy, relief, sadness) was 0.440. Sadness had the greatest rate, 0.64, anger the smallest, 0.11.
---
paper_title: Automatic Temporal Segment Detection and Affect Recognition From Face and Body Display

paper_content:

Psychologists have long explored mechanisms with which humans recognize other humans' affective states from modalities, such as voice and face display. This exploration has led to the identification of the main mechanisms, including the important role played in the recognition process by the modalities' dynamics. Constrained by the human physiology, the temporal evolution of a modality appears to be well approximated by a sequence of temporal segments called onset, apex, and offset. Stemming from these findings, computer scientists, over the past 15 years, have proposed various methodologies to automate the recognition process. We note, however, two main limitations to date. The first is that much of the past research has focused on affect recognition from single modalities. The second is that even the few multimodal systems have not paid sufficient attention to the modalities' dynamics: The automatic determination of their temporal segments, their synchronization to the purpose of modality fusion, and their role in affect recognition are yet to be adequately explored. To address this issue, this paper focuses on affective face and body display, proposes a method to automatically detect their temporal segments or phases, explores whether the detection of the temporal phases can effectively support recognition of affective states, and recognizes affective states based on phase synchronization/alignment. The experimental results obtained show the following: 1) affective face and body displays are simultaneous but not strictly synchronous; 2) explicit detection of the temporal phases can improve the accuracy of affect recognition; 3) recognition from fused face and body modalities performs better than that from the face or the body modality alone; and 4) synchronized feature-level fusion achieves better performance than decision-level fusion.
---
paper_title: Recognizing expressions from face and body gesture by temporal normalized motion and appearance features

paper_content:

Recently, recognizing affects from both face and body gestures attracts more attentions. However, it still lacks of efficient and effective features to describe the dynamics of face and gestures for real-time automatic affect recognition. In this paper, we propose a novel approach, which combines both MHI-HOG and Image-HOG through temporal normalization method, to describe the dynamics of face and body gestures for affect recognition. The MHI-HOG stands for Histogram of Oriented Gradients (HOG) on the Motion History Image (MHI). It captures motion direction of an interest point as an expression evolves over the time. The Image-HOG captures the appearance information of the corresponding interesting point. Combination of MHI-HOG and Image-HOG can effectively represent both local motion and appearance information of face and body gesture for affect recognition. The temporal normalization method explicitly solves the time resolution issue in the video-based affect recognition. Experimental results demonstrate promising performance as compared with the state of the art. We also show that expression recognition with temporal dynamics outperforms frame-based recognition.
---
paper_title: Recognising Human Emotions from Body Movement and Gesture Dynamics

paper_content:

We present an approach for the recognition of acted emotional states based on the analysis of body movement and gesture expressivity. According to research showing that distinct emotions are often associated with different qualities of body movement, we use non- propositional movement qualities (e.g. amplitude, speed and fluidity of movement) to infer emotions, rather than trying to recognise different gesture shapes expressing specific emotions. We propose a method for the analysis of emotional behaviour based on both direct classification of time series and a model that provides indicators describing the dynamics of expressive motion cues. Finally we show and interpret the recognition rates for both proposals using different classification algorithms.
---
paper_title: Affect recognition from face and body: early fusion vs. late fusion

paper_content:

This paper presents an approach to automatic visual emotion recognition from two modalities: face and body. Firstly, individual classifiers are trained from individual modalities. Secondly, we fuse facial expression and affective body gesture information first at a feature-level, in which the data from both modalities are combined before classification, and later at a decision-level, in which we integrate the outputs of the monomodal systems by the use of suitable criteria. We then evaluate these two fusion approaches, in terms of performance over monomodal emotion recognition based on facial expression modality only. In the experiments performed the emotion classification using the two modalities achieved a better recognition accuracy outperforming the classification using the individual facial modality. Moreover, fusion at the feature-level proved better recognition than fusion at the decision-level.
---
paper_title: Audio-visual emotion recognition using Boltzmann Zippers

paper_content:

This paper presents a novel approach for automatic audio-visual emotion recognition. The audio and visual channels provide complementary information for human emotional states recognition, and we utilize Boltzmann Zippers as model-level fusion to learn intrinsic correlations between the different modalities. We extract effective audio and visual feature streams with different time scales and feed them to two Boltzmann chains respectively. The hidden units of two chains are interconnected. Second-order methods are applied to Boltzmann Zippers to speed up learning and pruning process. Experimental results on audio-visual emotion data collected in Wizard of Oz scenarios demonstrate our approach is promising and outperforms single modal HMM and conventional coupled HMM methods.
---
paper_title: Iterative Feature Normalization Scheme for Automatic Emotion Detection from Speech

paper_content:

The externalization of emotion is intrinsically speaker-dependent. A robust emotion recognition system should be able to compensate for these differences across speakers. A natural approach is to normalize the features before training the classifiers. However, the normalization scheme should not affect the acoustic differences between emotional classes. This study presents the iterative feature normalization (IFN) framework, which is an unsupervised front-end, especially designed for emotion detection. The IFN approach aims to reduce the acoustic differences, between the neutral speech across speakers, while preserving the inter-emotional variability in expressive speech. This goal is achieved by iteratively detecting neutral speech for each speaker, and using this subset to estimate the feature normalization parameters. Then, an affine transformation is applied to both neutral and emotional speech. This process is repeated till the results from the emotion detection system are consistent between consecutive iterations. The IFN approach is exhaustively evaluated using the IEMOCAP database and a data set obtained under free uncontrolled recording conditions with different evaluation configurations. The results show that the systems trained with the IFN approach achieve better performance than systems trained either without normalization or with global normalization.
---
paper_title: Emotion recognition based on human gesture and speech information using RT middleware

paper_content:

A bi-modal emotion recognition approach is proposed for recognition of four emotions that integrate information from gestures and speech. The outputs from two uni-modal emotion recognition systems based on affective speech and expressive gesture are fused on a decision level fusion by using weight criterion fusion and best probability plus majority vote fusion methods, and the performance of classifier which performs better than each uni-modal and is helpful in recognizing suitable emotions for communication situations. To validate the proposal, fifty Japanese words (or phrases) and 8 types of gestures that are recorded from five participants are used, and the emotion recognition rate increases up to 85.39%. The proposal is able to extent to using more than other modalities and useful in automatic emotion recognition system for human-robot communication.
---
paper_title: Feature Analysis and Evaluation for Automatic Emotion Identification in Speech

paper_content:

The definition of parameters is a crucial step in the development of a system for identifying emotions in speech. Although there is no agreement on which are the best features for this task, it is generally accepted that prosody carries most of the emotional information. Most works in the field use some kind of prosodic features, often in combination with spectral and voice quality parametrizations. Nevertheless, no systematic study has been done comparing these features. This paper presents the analysis of the characteristics of features derived from prosody, spectral envelope, and voice quality as well as their capability to discriminate emotions. In addition, early fusion and late fusion techniques for combining different information sources are evaluated. The results of this analysis are validated with experimental automatic emotion identification tests. Results suggest that spectral envelope features outperform the prosodic ones. Even when different parametrizations are combined, the late fusion of long-term spectral statistics with short-term spectral envelope parameters provides an accuracy comparable to that obtained when all parametrizations are combined.
---
paper_title: Speech Emotion Recognition Using Fourier Parameters

paper_content:

Recently, studies have been performed on harmony features for speech emotion recognition. It is found in our study that the first- and second-order differences of harmony features also play an important role in speech emotion recognition. Therefore, we propose a new Fourier parameter model using the perceptual content of voice quality and the first- and second-order differences for speaker-independent speech emotion recognition. Experimental results show that the proposed Fourier parameter (FP) features are effective in identifying various emotional states in speech signals. They improve the recognition rates over the methods using Mel frequency cepstral coefficient (MFCC) features by 16.2, 6.8 and 16.6 points on the German database (EMODB), Chinese language database (CASIA) and Chinese elderly emotion database (EESDB). In particular, when combining FP with MFCC, the recognition rates can be further improved on the aforementioned databases by 17.5, 10 and 10.5 points, respectively.
---
paper_title: Learning Salient Features for Speech Emotion Recognition Using Convolutional Neural Networks

paper_content:

As an essential way of human emotional behavior understanding, speech emotion recognition (SER) has attracted a great deal of attention in human-centered signal processing. Accuracy in SER heavily depends on finding good affect- related , discriminative features. In this paper, we propose to learn affect-salient features for SER using convolutional neural networks (CNN). The training of CNN involves two stages. In the first stage, unlabeled samples are used to learn local invariant features (LIF) using a variant of sparse auto-encoder (SAE) with reconstruction penalization. In the second step, LIF is used as the input to a feature extractor, salient discriminative feature analysis (SDFA), to learn affect-salient, discriminative features using a novel objective function that encourages feature saliency, orthogonality, and discrimination for SER. Our experimental results on benchmark datasets show that our approach leads to stable and robust recognition performance in complex scenes (e.g., with speaker and language variation, and environment distortion) and outperforms several well-established SER features.
---
paper_title: Survey on speech emotion recognition: Features, classification schemes, and databases

paper_content:

Recently, increasing attention has been directed to the study of the emotional content of speech signals, and hence, many systems have been proposed to identify the emotional content of a spoken utterance. This paper is a survey of speech emotion classification addressing three important aspects of the design of a speech emotion recognition system. The first one is the choice of suitable features for speech representation. The second issue is the design of an appropriate classification scheme and the third issue is the proper preparation of an emotional speech database for evaluating system performance. Conclusions about the performance and limitations of current speech emotion recognition systems are discussed in the last section of this survey. This section also suggests possible ways of improving speech emotion recognition systems.
---
paper_title: Using a Smartphone to Measure Heart Rate Changes during Relived Happiness and Anger

paper_content:

This study demonstrates the feasibility of measuring heart rate (HR) differences associated with emotional states such as anger and happiness with a smartphone. Novice experimenters measured higher HRs during relived anger and happiness (replicating findings in the literature) outside a laboratory environment with a smartphone app that relied on photoplethysmography.
---
paper_title: Emotion Recognition Based on Multi-Variant Correlation of Physiological Signals

paper_content:

Emotion recognition based on affective physiological changes is a pattern recognition problem, and selecting specific physiological signals is necessary and helpful to recognize the emotions. Fingertip blood oxygen saturation (OXY), galvanic skin response (GSR) and heart rate (HR) are acquired while amusement, anger, grief and fear of 101 subjects are individually elicited by films. The affective physiological changes in multi-subject GSR, the first derivative of GSR (FD_GSR) and HR are detected by the multi-variant correlation method. The correlation analysis reveals that multi-subject HR, GSR and FD_GSR fluctuations respectively have common intra-class affective patterns. In addition to the conventional features of HR and GSR, the affective HR, GSR and FD_GSR fluctuations are quantified by the local scaling dimension and applied as the affective features. The multi-subject affective database containing 477 cases is classified by a Random Forests classifier. An overall correct rate of 74 percent for quinary classification of amusement, anger, grief, fear and the baseline state are obtained.
---
paper_title: Multimodal emotion recognition by combining physiological signals and facial expressions: A preliminary study

paper_content:

Lately, multimodal approaches for automatic emotion recognition have gained significant scientific interest. In this paper, emotion recognition by combining physiological signals and facial expressions was studied. Heart rate variability parameters, respiration frequency, and facial expressions were used to classify person's emotions while watching pictures with emotional content. Three classes were used for both valence and arousal. The preliminary results show that, over the proposed channels, detecting arousal seem to be easier compared to valence. While the classification performance of 54.5% was attained with arousal, only 38.0% of the samples were classified correctly in terms of valence. In future, additional modalities as well as feature selection will be utilized to improve the results.
---
paper_title: Robust EEG emotion classification using segment level decision fusion

paper_content:

In this paper we address single-trial binary classification of emotion dimensions (arousal, valence, dominance and liking) using electroencephalogram (EEG) signals that represent responses to audio-visual stimuli. We propose an innovative three step solution to this problem: (1) in contrast to the typical feature extraction on the response-level, we represent the EEG signal as a sequence of overlapping segments and extract feature vectors on the segment level; (2) transform segment level features to the response level features using projections based on a novel non-parametric nearest neighbor model; and (3) perform classification on the obtained response-level features. We demonstrate the efficacy of our approach by performing binary classification of emotion dimensions on DEAP (Dataset for Emotion Analysis using electroencephalogram, Physiological and Video Signals) and report state-of-the-art classification accuracies for all emotional dimensions.
---
paper_title: Multimodal emotion recognition using EEG and eye tracking data

paper_content:

This paper presents a new emotion recognition method which combines electroencephalograph (EEG) signals and pupillary response collected from eye tracker. We select 15 emotional film clips of 3 categories (positive, neutral and negative). The EEG signals and eye tracking data of five participants are recorded, simultaneously, while watching these videos. We extract emotion-relevant features from EEG signals and eye tracing data of 12 experiments and build a fusion model to improve the performance of emotion recognition. The best average accuracies based on EEG signals and eye tracking data are 71.77% and 58.90%, respectively. We also achieve average accuracies of 73.59% and 72.98% for feature level fusion strategy and decision level fusion strategy, respectively. These results show that both feature level fusion and decision level fusion combining EEG signals and eye tracking data can improve the performance of emotion recognition model.
---
paper_title: Affective Assessment by Digital Processing of the Pupil Diameter

paper_content:

Previous research found that the pupil diameter (PD) can be an indication of affective state, but this approach to the detection of the affective state of a computer user has not been investigated fully. We propose a new affective sensing approach to evaluate the computer user's affective states as they transition from "relaxationa#x201D; to "stress,a#x201D; through processing the PD signal. Wavelet denoising and Kalman filtering were used to preprocess the PD signal. Then, three features were extracted from it and five classification algorithms were used to evaluate the overall performance of the identification of "stressa#x201D; states in the computer users, achieving an average accuracy of 83.16 percent, with the highest accuracy of 84.21 percent reached with a Multilayer Perceptron and a Naive Bayes classifier. The Galvanic Skin Response (GSR) signal was also analyzed to study the comparative efficiency of affective sensing through the PD signal. We compared the discriminating power of the three features derived from the preprocessed PD signal to three features derived from the preprocessed GSR signal in terms of their Receiver Operating Characteristic curves. The results confirm that the PD signal should be considered a powerful physiological factor to involve in future automated affective classification systems for human-computer interaction.
---
paper_title: Exploring Fusion Methods for Multimodal Emotion Recognition with Missing Data

paper_content:

The study at hand aims at the development of a multimodal, ensemble-based system for emotion recognition. Special attention is given to a problem often neglected: missing data in one or more modalities. In offline evaluation the issue can be easily solved by excluding those parts of the corpus where one or more channels are corrupted or not suitable for evaluation. In real applications, however, we cannot neglect the challenge of missing data and have to find adequate ways to handle it. To address this, we do not expect examined data to be completely available at all time in our experiments. The presented system solves the problem at the multimodal fusion stage, so various ensemble techniques-covering established ones as well as rather novel emotion specific approaches-will be explained and enriched with strategies on how to compensate for temporarily unavailable modalities. We will compare and discuss advantages and drawbacks of fusion categories and extensive evaluation of mentioned techniques is carried out on the CALLAS Expressivity Corpus, featuring facial, vocal, and gestural modalities.
---
paper_title: The Faces of Engagement: Automatic Recognition of Student Engagementfrom Facial Expressions

paper_content:

Student engagement is a key concept in contemporary education, where it is valued as a goal in its own right. In this paper we explore approaches for automatic recognition of engagement from students’ facial expressions. We studied whether human observers can reliably judge engagement from the face; analyzed the signals observers use to make these judgments; and automated the process using machine learning. We found that human observers reliably agree when discriminating low versus high degrees of engagement (Cohen’s $\kappa = 0.96$ ). When fine discrimination is required (four distinct levels) the reliability decreases, but is still quite high ( $\kappa = 0.56$ ). Furthermore, we found that engagement labels of 10-second video clips can be reliably predicted from the average labels of their constituent frames (Pearson $r=0.85$ ), suggesting that static expressions contain the bulk of the information used by observers. We used machine learning to develop automatic engagement detectors and found that for binary classification (e.g., high engagement versus low engagement), automated engagement detectors perform with comparable accuracy to humans. Finally, we show that both human and automatic engagement judgments correlate with task performance. In our experiment, student post-test performance was predicted with comparable accuracy from engagement labels ( $r=0.47$ ) as from pre-test scores ( $r=0.44$ ).
---
paper_title: Automatically Recognizing Facial Expression: Predicting Engagement and Frustration

paper_content:

Learning involves a rich array of cognitive and affective states. Recognizing and understanding these cognitive and affective dimensions of learning is key to designing informed interventions. Prior research has highlighted the importance of facial expressions in learning-centered affective states, but tracking facial expression poses significant challenges. This paper presents an automated analysis of fine-grained facial movements that occur during computer-mediated tutoring. We use the Computer Expression Recognition Toolbox (CERT) to track fine-grained facial movements consisting of eyebrow raising (inner and outer), brow lowering, eyelid tightening, and mouth dimpling within a naturalistic video corpus of tutorial dialogue (N=65). Within the dataset, upper face movements were found to be predictive of engagement, frustration, and learning, while mouth dimpling was a positive predictor of learning and self-reported performance. These results highlight how both intensity and frequency of facial expressions predict tutoring outcomes. Additionally, this paper presents a novel validation of an automated tracking tool on a naturalistic tutoring dataset, comparing CERT results with manual annotations across a prior video corpus. With the advent of readily available fine-grained facial expression recognition, the developments introduced here represent a next step toward automatically understanding moment-by-moment affective states during learning.
---
paper_title: Affective e-Learning: Using "Emotional" Data to Improve Learning in Pervasive Learning Environment

paper_content:

Using emotion detection technologies from biophysical signals, this study explored how emotion evolves during learning process and how emotion feedback could be used to improve learning experiences. This article also described a cutting-edge pervasive e-Learning platform used in a Shanghai online college and proposed an affective e-Learning model, which combined learners’ emotions with the Shanghai e-Learning platform. The study was guided by Russell’s circumplex model of affect and Kort’s learning spiral model. The results about emotion recognition from physiological signals achieved a best-case accuracy (86.3%) for four types of learning emotions. And results from emotion revolution study showed that engagement and confusion were the most important and frequently occurred emotions in learning, which is consistent with the findings from AutoTutor project. No evidence from this study validated Kort’s learning spiral model. An experimental prototype of the affective e-Learning model was built to help improve students’ learning experience by customizing learning material delivery based on students’ emotional state. Experiments indicated the superiority of emotion aware over non-emotion-aware with a performance increase of 91%.
---
paper_title: Physiological signals of autistic children can be useful

paper_content:

This article covers the latest research concerning the measurement of physiological signals of children with autism, particularly for the study of changing emotions in various environments. Answers to important questions regarding autistic children's physiological activity are examined, and we will see that within a non-social environment, physiological responses are the same between children with and without autism but different in environments with social contexts. Moreover, physiological signals can be used as a reliable indicator of emotions of children with autism. Also covered are the latest developments in wearable sensor technologies avail- able for measuring on-the-go. I review additional research that identifies body signals in response to stimuli and may help ex- plain core social deficits in children with autism.
---
paper_title: The emotional hearing aid: an assistive tool for children with Asperger syndrome

paper_content:

People diagnosed along the autistic spectrum often have difficulties interacting with others in natural social environments. The emotional hearing aid is a portable assistive computer-based technology designed to help children with Asperger syndrome read and respond to the facial expressions of people they interact with. The tool implements the two principal elements that constitute one’s ability to empathize with others: the ability to identify a person’s mental state, a process known as mind-reading or theory of mind, and the ability to react appropriately to it (known as sympathizing). An automated mind-reading system attributes a mental state to a person by observing the behaviour of that person in real-time. Then the reaction advisor suggests to the user of the emotional hearing an appropriate reaction to the recognized mental state. This paper describes progress in the development and validation of the emotional hearing aid on two fronts. First, the implementation of the reaction advisor is described, showing how it takes into account the persistence, intensity and degree of confidence of a mental state inference. Second, the paper presents an experimental evaluation of the automated mind-reading system on six classes of complex mental states. In light of this progress, the paper concludes with a discussion of the challenges that still need to be addressed in developing and validating the emotional hearing aid.
---
paper_title: Intelligent Facial Action and emotion recognition for humanoid robots

paper_content:

This research focuses on the development of a realtime intelligent facial emotion recognition system for a humanoid robot. In our system, Facial Action Coding System is used to guide the automatic analysis of emotional facial behaviours. The work includes both an upper and a lower facial Action Units (AU) analyser. The upper facial analyser is able to recognise six AUs including Inner and Outer Brow Raiser, Upper Lid Raiser etc, while the lower facial analyser is able to detect eleven AUs including Upper Lip Raiser, Lip Corner Puller, Chin Raiser, etc. Both of the upper and lower analysers are implemented using feedforward Neural Networks (NN). The work also further decodes six basic emotions from the recognised AUs. Two types of facial emotion recognisers are implemented, NN-based and multi-class Support Vector Machine (SVM) based. The NN-based facial emotion recogniser with the above recognised AUs as inputs performs robustly and efficiently. The Multi-class SVM with the radial basis function kernel enables the robot to outperform the NN-based emotion recogniser in real-time posed facial emotion detection tasks for diverse testing subjects.
---
paper_title: Emotion and Gesture Recognition with Soft Computing Tool for Drivers Assistance System in Human Centered Transportation

paper_content:

Expression recognition or Emotional state recognition using holistic and feature information is the vital step in Driver Assistance System. Many researchers have work on Facial Gesture or Emotion recognition independently. The purpose of the present paper is to deal with Simultaneous Facial Gesture tracking and Emotion recognition with Soft Computing tool like Fuzzy rule based system (FBS). In Human Centered Transportation large number of road accidents took place due to drowsiness or bad mood of the driver. The system proposed in this paper take into account both the Facial Gesture tracking and Emotion recognition so that if there is any sign of less attentiveness of the driver or driver's fatigue the car will be switch to automatic mode. A novel fuzzy system is created, whose rules is being defined through analysis of Facial Gesture variations. The idea behind this paper is to detect Facial Gesture by detecting the motion of eyes & lips along with classification of different facial expressions into one of the four basic human emotions, viz. happy, anger, sad, and surprise with fuzzy rule based system for better system performance. The given system proposes 91.66% accuracy for Facial Gesture detection & 90% accuracy for Emotion recognition while using Simultaneous Facial Gesture detection and Emotion recognition it provides 94.58% accuracy.
---
paper_title: Smart recognition and synthesis of emotional speech for embedded systems with natural user interfaces

paper_content:

The importance of the emotion information in human speech has been growing in recent years due to increasing use of natural user interfacing in embedded systems. Speech-based human-machine communication has the advantage of a high degree of usability, but it need not be limited to speech-to-text and text-to-speech capabilities. Emotion recognition in uttered speech has been considered in this research to integrate a speech recognizer/synthesizer with the capacity to recognize and synthesize emotion. This paper describes a complete framework for recognizing and synthesizing emotional speech based on smart logic (fuzzy logic and artificial neural networks). Time-domain signal-processing algorithms has been applied to reduce computational complexity at the feature-extraction level. A fuzzy-logic engine was modeled to make inferences about the emotional content of the uttered speech. An artificial neural network was modeled to synthesize emotive speech. Both were designed to be integrated into an embedded handheld device that implements a speech-based natural user interface (NUI).
---
paper_title: Directing Physiology and Mood through Music: Validation of an Affective Music Player

paper_content:

Music is important in everyday life, as it provides entertainment and influences our moods. As music is widely available, it is becoming increasingly difficult to select songs to suit our mood. An affective music player can remove this obstacle by taking a desired mood as input and then selecting songs that direct toward that desired mood. In the present study, we validate the concept of an affective music player directing the energy dimension of mood. User models were trained for 10 participants based on skin conductance changes to songs from their own music database. Based on the resulting user models, the songs that most increased or decreased the skin conductance level of the participants were selected to induce either a relatively energized or a calm mood. Experiments were conducted in a real-world office setting. The results showed that a reliable prediction can be made of the impact of a song on skin conductance, that skin conductance and mood can be directed toward an energized or calm state and that skin conductance remains in these states for at least 30 minutes. All in all, this study shows that the concept and models of the affective music player worked in an ecologically valid setting, suggesting the feasibility of using physiological responses in real-life affective computing applications.
---
paper_title: Emotion Assessment From Physiological Signals for Adaptation of Game Difficulty

paper_content:

This paper proposes to maintain player's engagement by adapting game difficulty according to player's emotions assessed from physiological signals. The validity of this approach was first tested by analyzing the questionnaire responses, electroencephalogram (EEG) signals, and peripheral signals of the players playing a Tetris game at three difficulty levels. This analysis confirms that the different difficulty levels correspond to distinguishable emotions, and that, playing several times at the same difficulty level gives rise to boredom. The next step was to train several classifiers to automatically detect the three emotional classes from EEG and peripheral signals in a player-independent framework. By using either type of signals, the emotional classes were successfully recovered, with EEG having a better accuracy than peripheral signals on short periods of time. After the fusion of the two signal categories, the accuracy raised up to 63%.
---
paper_title: A study on human age estimation under facial expression changes

paper_content:

In this paper, we study human age estimation in face images under significant expression changes. We will address two issues: (1) Is age estimation affected by facial expression changes and how significant is the influence? (2) How to develop a robust method to perform age estimation undergoing various facial expression changes? This systematic study will not only discover the relation between age estimation and expression changes, but also contribute a robust solution to solve the problem of cross-expression age estimation. This study is an important step towards developing a practical and robust age estimation system that allows users to present their faces naturally (with various expressions) rather than constrained to the neutral expression only. Two databases originally captured in the Psychology community are introduced to Computer Vision, to quantitatively demonstrate the influence of expression changes on age estimation, and evaluate the proposed framework and corresponding methods for cross-expression age estimation.
---
paper_title: Exploring Cross-Modality Affective Reactions for Audiovisual Emotion Recognition

paper_content:

Psycholinguistic studies on human communication have shown that during human interaction individuals tend to adapt their behaviors mimicking the spoken style, gestures, and expressions of their conversational partners. This synchronization pattern is referred to as entrainment. This study investigates the presence of entrainment at the emotion level in cross-modality settings and its implications on multimodal emotion recognition systems. The analysis explores the relationship between acoustic features of the speaker and facial expressions of the interlocutor during dyadic interactions. The analysis shows that 72 percent of the time the speakers displayed similar emotions, indicating strong mutual influence in their expressive behaviors. We also investigate the cross-modality, cross-speaker dependence, using mutual information framework. The study reveals a strong relation between facial and acoustic features of one subject with the emotional state of the other subject. It also shows strong dependence between heterogeneous modalities across conversational partners. These findings suggest that the expressive behaviors from one dialog partner provide complementary information to recognize the emotional state of the other dialog partner. The analysis motivates classification experiments exploiting cross-modality, cross-speaker information. The study presents emotion recognition experiments using the IEMOCAP and SEMAINE databases. The results demonstrate the benefit of exploiting this emotional entrainment effect, showing statistically significant improvements.
---
paper_title: Comparison of Gender- and Speaker-adaptive Emotion Recognition

paper_content:

Deriving the emotion of a human speaker is a hard task, especially if only the audio stream is taken into account. While state-of-the-art approaches already provide good results, adaptive methods have been proposed in order to further improve the recognition accuracy. A recent approach is to add characteristics of the speaker, e.g., the gender of the speaker. In this contribution, we argue that adding information unique for each speaker, i.e., by using speaker identification techniques, improves emotion recognition simply by adding this additional information to the feature vector of the statistical classification algorithm. Moreover, we compare this approach to emotion recognition adding only the speaker gender being a non-unique speaker attribute. We justify this by performing adaptive emotion recognition using both gender and speaker information on four different corpora of different languages containing acted and non-acted speech. The final results show that adding speaker information significantly outperforms both adding gender information and solely using a generic speaker-independent approach.
---
paper_title: Tracking changes in continuous emotion states using body language and prosodic cues

paper_content:

Human expressive interactions are characterized by an ongoing unfolding of verbal and nonverbal cues. Such cues convey the interlocutor's emotional state which is continuous and of variable intensity and clarity over time. In this paper, we examine the emotional content of body language cues describing a participant's posture, relative position and approach/withdraw behaviors during improvised affective interactions, and show that they reflect changes in the participant's activation and dominance levels. Furthermore, we describe a framework for tracking changes in emotional states during an interaction using a statistical mapping between the observed audiovisual cues and the underlying user state. Our approach shows promising results for tracking changes in activation and dominance.
---
paper_title: Facial Expression Recognition Influenced by Human Aging

paper_content:

Facial expression recognition (FER) is an active research topic in computer vision. However, there is no study yet to discover whether FER is affected by human aging, from a computational perspective. We perform a computational study of FER within and across age groups and compare the FER accuracies. Two databases from the psychology society are introduced to the computer vision community and used for our study. We found that the FER is influenced significantly by human aging, and we analyze the influence and interpret it from a computational viewpoint. Next, we propose some schemes to reduce the influence of aging on FER and evaluate the effectiveness in dealing with lifespan FER.
---
paper_title: Context-Sensitive Dynamic Ordinal Regression for Intensity Estimation of Facial Action Units

paper_content:

Modeling intensity of facial action units from spontaneously displayed facial expressions is challenging mainly because of high variability in subject-specific facial expressiveness, head-movements, illumination changes, etc. These factors make the target problem highly context-sensitive. However, existing methods usually ignore this context-sensitivity of the target problem. We propose a novel Conditional Ordinal Random Field (CORF) model for context-sensitive modeling of the facial action unit intensity, where the W5+ ( who , when , what , where , why and how ) definition of the context is used. While the proposed model is general enough to handle all six context questions, in this paper we focus on the context questions: who (the observed subject), how (the changes in facial expressions), and when (the timing of facial expressions and their intensity). The context questions who and how are modeled by means of the newly introduced context-dependent covariate effects, and the context question when is modeled in terms of temporal correlation between the ordinal outputs, i.e., intensity levels of action units. We also introduce a weighted softmax-margin learning of CRFs from data with skewed distribution of the intensity levels, which is commonly encountered in spontaneous facial data. The proposed model is evaluated on intensity estimation of pain and facial action units using two recently published datasets (UNBC Shoulder Pain and DISFA) of spontaneously displayed facial expressions. Our experiments show that the proposed model performs significantly better on the target tasks compared to the state-of-the-art approaches. Furthermore, compared to traditional learning of CRFs, we show that the proposed weighted learning results in more robust parameter estimation from the imbalanced intensity data.
---
paper_title: Context-Sensitive Learning for Enhanced Audiovisual Emotion Classification

paper_content:

Human emotional expression tends to evolve in a structured manner in the sense that certain emotional evolution patterns, i.e., anger to anger, are more probable than others, e.g., anger to happiness. Furthermore, the perception of an emotional display can be affected by recent emotional displays. Therefore, the emotional content of past and future observations could offer relevant temporal context when classifying the emotional content of an observation. In this work, we focus on audio-visual recognition of the emotional content of improvised emotional interactions at the utterance level. We examine context-sensitive schemes for emotion recognition within a multimodal, hierarchical approach: bidirectional Long Short-Term Memory (BLSTM) neural networks, hierarchical Hidden Markov Model classifiers (HMMs), and hybrid HMM/BLSTM classifiers are considered for modeling emotion evolution within an utterance and between utterances over the course of a dialog. Overall, our experimental results indicate that incorporating long-term temporal context is beneficial for emotion recognition systems that encounter a variety of emotional manifestations. Context-sensitive approaches outperform those without context for classification tasks such as discrimination between valence levels or between clusters in the valence-activation space. The analysis of emotional transitions in our database sheds light into the flow of affective expressions, revealing potentially useful patterns.
---

Title: A Survey on Human Emotion Recognition Approaches, Databases and Applications

Section 1: Introduction
Description 1: This section introduces the field of emotion recognition, its significance in AI and Human-Computer Interaction, and the survey's structure.

Section 2: Emotions and affective computing
Description 2: This section explains the various theories of emotion and the interdisciplinary field of Affective Computing (AC).

Section 3: Emotion recognition modalities and approaches
Description 3: This section discusses the different modalities (facial expressions, bodily expressions, speech signals, physiological signals) and approaches used in emotion recognition systems.

Section 4: Facial Expressions
Description 4: This section focuses on the use of facial expressions in emotion recognition, including methodologies and challenges.

Section 5: Bodily expressions
Description 5: This section explains the role and methodologies involving body gestures for emotion recognition.

Section 6: Speech Signals
Description 6: This section outlines how verbal communication and audio features are used to recognize emotions.

Section 7: Physiological signals
Description 7: This section reviews approaches that use physiological signals like heart rate and galvanic skin response for emotion recognition.

Section 8: Other Modalities
Description 8: This section discusses less common or emerging modalities and their application in emotion recognition systems.

Section 9: Emotion recognition databases
Description 9: This section provides an overview of various databases used in emotion recognition research, categorized by types such as image, video, speech, and multimodal databases.

Section 10: Emotion recognition Applications
Description 10: This section outlines the diverse application areas of emotion recognition systems, including education, aid for the disabled, HCI/robotics, safety aids, and entertainment.

Section 11: Conclusion and Future Directions
Description 11: This section summarizes the survey and provides insights into future research directions and challenges in emotion recognition.