Survey on AI-Based Multimodal Methods for Emotion Detection
---
paper_title: Inference of attitudes from nonverbal communication in two channels.

paper_content:

Hydroxy terminated polybutadiene is reacted with naphtyl-potassium and 1-bromo-2,4-pentadiene in successive steps to form a bis(1,3-pentadienyl ether) derivative. Then the bismaleimide of dimer diamine is added to the polybutadiene derivative whereby a room temperature cure to an elastomer is achieved.
---
paper_title: AutoEmotive: bringing empathy to the driving experience to manage stress

paper_content:

With recent developments in sensing technologies, it's becoming feasible to comfortably measure several aspects of emotions during challenging daily life situations. This work describes how the stress of drivers can be measured through different types of interactions, and how the information can enable several interactions in the car with the goal of helping to manage stress. These new interactions could help not only to bring empathy to the driving experience but also to improve driver safety and increase social awareness.
---
paper_title: Toward Machine Emotional Intelligence: Analysis of Affective Physiological State

paper_content:

The ability to recognize emotion is one of the hallmarks of emotional intelligence, an aspect of human intelligence that has been argued to be even more important than mathematical and verbal intelligences. This paper proposes that machine intelligence needs to include emotional intelligence and demonstrates results toward this goal: developing a machine's ability to recognize the human affective state given four physiological signals. We describe difficult issues unique to obtaining reliable affective data and collect a large set of data from a subject trying to elicit and experience each of eight emotional states, daily, over multiple weeks. This paper presents and compares multiple algorithms for feature-based recognition of emotional state from this data. We analyze four physiological signals that exhibit problematic day-to-day variations: The features of different emotions on the same day tend to cluster more tightly than do the features of the same emotion on different days. To handle the daily variations, we propose new features and algorithms and compare their performance. We find that the technique of seeding a Fisher Projection with the results of sequential floating forward search improves the performance of the Fisher Projection and provides the highest recognition rates reported to date for classification of affect from physiology: 81 percent recognition accuracy on eight classes of emotion, including neutral.
---
paper_title: YouTube Movie Reviews: Sentiment Analysis in an Audio-Visual Context

paper_content:

This work focuses on automatically analyzing a speaker's sentiment in online videos containing movie reviews. In addition to textual information, this approach considers adding audio features as typically used in speech-based emotion recognition as well as video features encoding valuable valence information conveyed by the speaker. Experimental results indicate that training on written movie reviews is a promising alternative to exclusively using (spoken) in-domain data for building a system that analyzes spoken movie review videos, and that language-independent audio-visual analysis can compete with linguistic analysis.
---
paper_title: Multimodal Sentiment Intensity Analysis in Videos: Facial Gestures and Verbal Messages

paper_content:

People share their opinions, stories, and reviews through online video sharing websites every day. The automatic analysis of these online opinion videos is bringing new or understudied research challenges to the field of computational linguistics and multimodal analysis. Among these challenges is the fundamental question of exploiting the dynamics between visual gestures and verbal messages to be able to better model sentiment. This article addresses this question in four ways: introducing the first multimodal dataset with opinion-level sentiment intensity annotations; studying the prototypical interaction patterns between facial gestures and spoken words when inferring sentiment intensity; proposing a new computational representation, called multimodal dictionary, based on a language-gesture study; and evaluating the authors' proposed approach in a speaker-independent paradigm for sentiment intensity prediction. The authors' study identifies four interaction types between facial gestures and verbal content: neutral, emphasizer, positive, and negative interactions. Experiments show statistically significant improvement when using multimodal dictionary representation over the conventional early fusion representation (that is, feature concatenation).
---
paper_title: Tensor Fusion Network for Multimodal Sentiment Analysis

paper_content:

Multimodal sentiment analysis is an increasingly popular research area, which extends the conventional language-based definition of sentiment analysis to a multimodal setup where other relevant modalities accompany language. In this paper, we pose the problem of multimodal sentiment analysis as modeling intra-modality and inter-modality dynamics. We introduce a novel model, termed Tensor Fusion Network, which learns both such dynamics end-to-end. The proposed approach is tailored for the volatile nature of spoken language in online videos as well as accompanying gestures and voice. In the experiments, our model outperforms state-of-the-art approaches for both multimodal and unimodal sentiment analysis.
---
paper_title: Fusing audio, visual and textual clues for sentiment analysis from multimodal content

paper_content:

A huge number of videos are posted every day on social media platforms such as Facebook and YouTube. This makes the Internet an unlimited source of information. In the coming decades, coping with such information and mining useful knowledge from it will be an increasingly difficult task. In this paper, we propose a novel methodology for multimodal sentiment analysis, which consists in harvesting sentiments from Web videos by demonstrating a model that uses audio, visual and textual modalities as sources of information. We used both feature- and decision-level fusion methods to merge affective information extracted from multiple modalities. A thorough comparison with existing works in this area is carried out throughout the paper, which demonstrates the novelty of our approach. Preliminary comparative experiments with the YouTube dataset show that the proposed multimodal system achieves an accuracy of nearly 80%, outperforming all state-of-the-art systems by more than 20%.
---
paper_title: Utterance-Level Multimodal Sentiment Analysis

paper_content:

During real-life interactions, people are naturally gesturing and modulating their voice to emphasize specific points or to express their emotions. With the recent growth of social websites such as YouTube, Facebook, and Amazon, video reviews are emerging as a new source of multimodal and natural opinions that has been left almost untapped by automatic opinion analysis techniques. This paper presents a method for multimodal sentiment classification, which can identify the sentiment expressed in utterance-level visual datastreams. Using a new multimodal dataset consisting of sentiment annotated utterances extracted from video reviews, we show that multimodal sentiment analysis can be effectively performed, and that the joint use of visual, acoustic, and linguistic modalities can lead to error rate reductions of up to 10.5% as compared to the best performing individual modality.
---
paper_title: RRSS-Rating Reviews Support System purpose built for movies recommendation

paper_content:

This paper describes the part of a recommendation system designed for the recognition of film reviews (RRSS). Such a system allows the automatic collection, evaluation and rating of reviews and opinions of the movies. First the system searches and retrieves texts supposed to be movie reviews from the Internet. Subsequently the system carries out an evaluation and rating of the movie reviews. Finally, the system automatically associates a digital assessment with each review. The goal of the system is to give the score of reviews associated with the user who wrote them. All of this data is the input to the cognitive engine. Data from our base allows the making of correspondences, which are required for cognitive algorithms to improve, advanced recommending functionalities for e-business and e-purchase websites. In this paper we will describe the different methods on automatically identifying opinions using natural language knowledge and techniques of classification.
---
paper_title: Tool of the intelligence economic: Recognition function of reviews critics

paper_content:

This paper describes the part of recommender system designed for movies’ critics recognition. Such a system allows the automatic collection, evaluation and rating of critics and opinions of the movies. First the system searches and retrieves texts supposed to be movies’ reviews from the Internet. Subsequently the system carries out an evaluation and rating of movies’ critics. Finally the system automatically associates a numerical mark to each critic. The goal of system is to give the score of critics associated to the users’ who wrote them. All of this data are the input to the cognitive engine. Data from our base allow making correspondences which are required for cognitive algorithms to improve advanced recommending functionalities for e-business and e-purchases websites. Our sesystem uses three different methods for classifying opinions from reviews critics. In this paper we describe the part of system which is based on automatically identifying opinions using natural language processing knowledge.
---
paper_title: Opinion mining and sentiment analysis

paper_content:

An important part of our information-gathering behavior has always been to find out what other people think. With the growing availability and popularity of opinion-rich resources such as online review sites and personal blogs, new opportunities and challenges arise as people now can, and do, actively use information technologies to seek out and understand the opinions of others. The sudden eruption of activity in the area of opinion mining and sentiment analysis, which deals with the computational treatment of opinion, sentiment, and subjectivity in text, has thus occurred at least in part as a direct response to the surge of interest in new systems that deal directly with opinions as a first-class object. ::: ::: This survey covers techniques and approaches that promise to directly enable opinion-oriented information-seeking systems. Our focus is on methods that seek to address the new challenges raised by sentiment-aware applications, as compared to those that are already present in more traditional fact-based analysis. We include material on summarization of evaluative text and on broader issues regarding privacy, manipulation, and economic impact that the development of opinion-oriented information-access services gives rise to. To facilitate future work, a discussion of available resources, benchmark datasets, and evaluation campaigns is also provided.
---
paper_title: EXPLORE THE EFFECTS OF EMOTICONS ON TWITTER SENTIMENT ANALYSIS

paper_content:

In recent years, Twitter Sentiment Analysis (TSA) has become a hot research topic. The target of this task is to analyse the sentiment polarity of the tweets. There are a lot of machine learning methods specifically developed to solve TSA problems, such as fully supervised method, distantly supervised method and combined method of these two. Considering the specialty of tweets that a limitation of 140 characters, emoticons have important effects on TSA. In this paper, we compare three emoticon pre-processing methods: emotion deletion (emoDel), emoticons 2-valued translation (emo2label) and emoticon explanation (emo2explanation). Then, we propose a method based on emoticon-weight lexicon, and conduct experiments based on Naive Bayes classifier, to validate the crucial role emoticons play on guiding emotion tendency in a tweet. Experiments on real data sets demonstrate that emoticons are vital to TSA.
---
paper_title: Sentiment analysis on tweets for social events

paper_content:

Sentiment analysis or opinion mining is an important type of text analysis that aims to support decision making by extracting and analyzing opinion oriented text, identifying positive and negative opinions, and measuring how positively or negatively an entity (i.e., people, organization, event, location, product, topic, etc.) is regarded. As more and more users express their political and religious views on Twitter, tweets become valuable sources of people's opinions. Tweets data can be efficiently used to infer people's opinions for marketing or social studies. This paper proposes a Tweets Sentiment Analysis Model (TSAM) that can spot the societal interest and general people's opinions in regard to a social event. In this paper, Australian federal election 2010 event was taken as an example for sentiment analysis experiments. We are primarily interested in the sentiment of the specific political candidates, i.e., two primary minister candidates - Julia Gillard and Tony Abbot. Our experimental results demonstrate the effectiveness of the system.
---
paper_title: Twitter brand sentiment analysis: A hybrid system using n-gram analysis and dynamic artificial neural network

paper_content:

Twitter messages are increasingly used to determine consumer sentiment towards a brand. The existing literature on Twitter sentiment analysis uses various feature sets and methods, many of which are adapted from more traditional text classification problems. In this research, we introduce an approach to supervised feature reduction using n-grams and statistical analysis to develop a Twitter-specific lexicon for sentiment analysis. We augment this reduced Twitter-specific lexicon with brand-specific terms for brand-related tweets. We show that the reduced lexicon set, while significantly smaller (only 187 features), reduces modeling complexity, maintains a high degree of coverage over our Twitter corpus, and yields improved sentiment classification accuracy. To demonstrate the effectiveness of the devised Twitter-specific lexicon compared to a traditional sentiment lexicon, we develop comparable sentiment classification models using SVM. We show that the Twitter-specific lexicon is significantly more effective in terms of classification recall and accuracy metrics. We then develop sentiment classification models using the Twitter-specific lexicon and the DAN2 machine learning approach, which has demonstrated success in other text classification problems. We show that DAN2 produces more accurate sentiment classification results than SVM while using the same Twitter-specific lexicon.
---
paper_title: Emoticon Smoothed Language Models for Twitter Sentiment Analysis

paper_content:

Twitter sentiment analysis (TSA) has become a hot research topic in recent years. The goal of this task is to discover the attitude or opinion of the tweets, which is typically formulated as a machine learning based text classification problem. Some methods use manually labeled data to train fully supervised models, while others use some noisy labels, such as emoticons and hashtags, for model training. In general, we can only get a limited number of training data for the fully supervised models because it is very labor-intensive and time-consuming to manually label the tweets. As for the models with noisy labels, it is hard for them to achieve satisfactory performance due to the noise in the labels although it is easy to get a large amount of data for training. Hence, the best strategy is to utilize both manually labeled data and noisy labeled data for training. However, how to seamlessly integrate these two different kinds of data into the same learning framework is still a challenge. In this paper, we present a novel model, called emoticon smoothed language model (ESLAM), to handle this challenge. The basic idea is to train a language model based on the manually labeled data, and then use the noisy emoticon data for smoothing. Experiments on real data sets demonstrate that ESLAM can effectively integrate both kinds of data to outperform those methods using only one of them.
---
paper_title: Predicting Depression From Language-Based Emotion Dynamics: Longitudinal Analysis of Facebook and Twitter Status Updates

paper_content:

Background: Frequent expression of negative emotion words on social media has been linked to depression. However, metrics have relied on average values, not dynamic measures of emotional volatility. Objective: The aim of this study was to report on the associations between depression severity and the variability (time-unstructured) and instability (time-structured) in emotion word expression on Facebook and Twitter across status updates. Methods: Status updates and depression severity ratings of 29 Facebook users and 49 Twitter users were collected through the app MoodPrism. The average proportion of positive and negative emotion words used, within-person variability, and instability were computed. Results: Negative emotion word instability was a significant predictor of greater depression severity on Facebook (rs(29)=.44, P=.02, 95% CI 0.09-0.69), even after controlling for the average proportion of negative emotion words used (partial rs(26)=.51, P=.006) and within-person variability (partial rs(26)=.49, P=.009). A different pattern emerged on Twitter where greater negative emotion word variability indicated lower depression severity (rs(49)=−.34, P=.01, 95% CI −0.58 to 0.09). Differences between Facebook and Twitter users in their emotion word patterns and psychological characteristics were also explored. Conclusions: The findings suggest that negative emotion word instability may be a simple yet sensitive measure of time-structured variability, useful when screening for depression through social media, though its usefulness may depend on the social media platform. [J Med Internet Res 2018;20(5):e168]
---
paper_title: Once More with Feeling: Supportive Responses to Social Sharing on Facebook

paper_content:

Life is more than cat pictures. There are tough days, heartbreak, and hugs. Under what contexts do people share these feelings online, and how do their friends respond? Using millions of de-identified Facebook status updates with poster-annotated feelings (e.g., “feeling thankful” or “feeling worried”), we examine the magnitude and circumstances in which people share positive or negative feelings and characterize the nature of the responses they receive. We find that people share greater proportions of both positive and negative emotions when their friend networks are smaller and denser. Consistent with social sharing theory, hearing about a friendâ s troubles on Facebook causes friends to reply with more emotional and supportive comments. Friendsâ comments are also more numerous and longer. Posts with positive feelings, on the other hand, receive more likes, and their comments have more positive language. Feelings that relate to the posterâ s self worth, such as “feeling defeated,” “feeling unloved,” or “feeling accomplished” amplify these effects.
---
paper_title: Twitter Analysis: Studying US Weekly Trends in Work Stress and Emotion

paper_content:

We propose the use of Twitter analysis as an alternative source of data to document weekly trends in emotion and stress, and attempt to use the method to estimate the work recovery effect of weekends. On the basis of 2,102,176,189 Tweets, we apply Pennebaker's linguistic inquiry word count (LIWC) approach to measure daily Tweet content across 18 months, aggregated to the US national level of analysis. We derived a word count dictionary to assess work stress and applied p-technique factor analysis to the daily word count data from 19 substantively different content areas covered by the LIWC dictionaries. Dynamic factor analysis revealed two latent factors in day-level variation of Tweet content. These two factors are: (a) a negative emotion/stress/somatic factor, and (b) a positive emotion/food/friends/home/family/leisure factor, onto which elements of work, money, achievement, and health issues have strong negative loadings. The weekly trend analysis revealed a clear “Friday dip” for work stress and negative emotion expressed on Twitter. In contrast, positive emotion Tweets showed a “mid-week dip” for Tuesday-Wednesday-Thursday and “weekend peak” for Friday through Sunday, whereas work/money/achievement/health problem Tweets showed a small “weekend dip” on Fridays through Sundays. Results partially support the Effort-Recovery theory. Implications and limitations of the method are discussed.
---
paper_title: When perceptions defy reality: The relationships between depression and actual and perceived Facebook social support

paper_content:

Abstract Background Although the relationship between depression and “offline” social support is well established, numerous questions surround the relationship between “online” social support and depression. We explored this issue by examining the social support dynamics that characterize the way individuals with varying levels of depression (Study 1) and SCID-diagnosed clinically depressed and non-depressed individuals (Study 2) interact with Facebook, the world's largest online social network. Method Using a novel methodology, we examined how disclosing positive or negative information on Facebook influences the amount of social support depressed individuals (a) actually receive (based on actual social support transactions recorded on Facebook walls) and (b) think they receive (based on subjective assessments) from their Facebook network. Results Contrary to prior research indicating that depression correlates with less actual social support from “offline” networks, across both studies depression was positively correlated with social support from Facebook networks when participants disclosed negative information ( p =.02 in Study 1 and p =.06 in Study 2). Yet, depression was negatively correlated with how much social support participants thought they received from their Facebook networks ( p =.005 in Study 1 and p =.001 in Study 2). Limitations The sample size was relatively small in Study 2, reflecting difficulties of recruiting individuals with Major Depressive Disorder. Conclusions These results demonstrate that an asymmetry characterizes the relationship between depression and different types of Facebook social support and further identify perceptions of Facebook social support as a potential intervention target. (243 words; 250 max)
---
paper_title: Short text classification in twitter to improve information filtering

paper_content:

In microblogging services such as Twitter, the users may become overwhelmed by the raw data. One solution to this problem is the classification of short text messages. As short texts do not provide sufficient word occurrences, traditional classification methods such as "Bag-Of-Words" have limitations. To address this problem, we propose to use a small set of domain-specific features extracted from the author's profile and text. The proposed approach effectively classifies the text to a predefined set of generic classes such as News, Events, Opinions, Deals, and Private Messages.
---
paper_title: The dynamics of health behavior sentiments on a large online social network

paper_content:

Modifiable health behaviors, a leading cause of illness and death in many countries, are often driven by individual beliefs and sentiments about health and disease. Individual behaviors affecting health outcomes are increasingly modulated by social networks, for example through the associations of like-minded individuals - homophily - or through peer influence effects. Using a statistical approach to measure the individual temporal effects of a large number of variables pertaining to social network statistics, we investigate the spread of a health sentiment towards a new vaccine on Twitter, a large online social network. We find that the effects of neighborhood size and exposure intensity are qualitatively very different depending on the type of sentiment. Generally, we find that larger numbers of opinionated neighbors inhibit the expression of sentiments. We also find that exposure to negative sentiment is contagious - by which we merely mean predictive of future negative sentiment expression - while exposure to positive sentiments is generally not. In fact, exposure to positive sentiments can even predict increased negative sentiment expression. Our results suggest that the effects of peer influence and social contagion on the dynamics of behavioral spread on social networks are strongly content-dependent.
---
paper_title: Fisher Kernels on Phase-Based Features for Speech Emotion Recognition

paper_content:

The involvement of affect information in a spoken dialogue system can increase the user-friendliness and provide a more natural way for the interaction experience. This can be reached by speech emotion recognition, where the features are usually dominated by the spectral amplitude information while they ignore the use of the phase spectrum. In this chapter, we propose to use phase-based features to build up such an emotion recognition system. To exploit these features, we employ Fisher kernels. The according technique encodes the phase-based features by their deviation from a generative Gaussian mixture model. The resulting representation is fed to train a classification model with a linear kernel classifier. Experimental results on the GeWEC database including ‘normal’ and whispered phonation demonstrate the effectiveness of our method.
---
paper_title: Comparative analysis of emotion estimation methods based on physiological measurements for real-time applications

paper_content:

In order to improve intelligent Human-Computer Interaction it is important to create a personalized adaptive emotion estimator that is able to learn over time emotional response idiosyncrasies of individual person and thus enhance estimation accuracy. This paper, with the aim of identifying preferable methods for such a concept, presents an experiment-based comparative study of seven feature reduction and seven machine learning methods commonly used for emotion estimation based on physiological signals. The analysis was performed on data obtained in an emotion elicitation experiment involving 14 participants. Specific discrete emotions were targeted with stimuli from the International Affective Picture System database. The experiment was necessary to achieve the uniformity in the various aspects of emotion elicitation, data processing, feature calculation, self-reporting procedures and estimation evaluation, in order to avoid inconsistency problems that arise when results from studies that use different emotion-related databases are mutually compared. The results of the performed experiment indicate that the combination of a multilayer perceptron (MLP) with sequential floating forward selection (SFFS) exhibited the highest accuracy in discrete emotion classification based on physiological features calculated from ECG, respiration, skin conductance and skin temperature. Using leave-one-session-out crossvalidation method, 60.3% accuracy in classification of 5 discrete emotions (sadness, disgust, fear, happiness and neutral) was obtained. In order to identify which methods may be the most suitable for real-time estimator adaptation, execution and learning times of emotion estimators were also comparatively analyzed. Based on this analysis, preferred feature reduction method for real-time estimator adaptation was minimum redundancy - maximum relevance (mRMR), which was the fastest approach in terms of combined execution and learning time, as well as the second best in accuracy, after SFFS. In combination with mRMR, highest accuracies were achieved by k-nearest neighbor (kNN) and MLP with negligible difference (50.33% versus 50.54%); however, mRMR+kNN is preferable option for real-time estimator adaptation due to considerably lower combined execution and learning time of kNN versus MLP.
---
paper_title: Analysis of emotion recognition using facial expressions, speech and multimodal information

paper_content:

The interaction between human beings and computers will be more natural if computers are able to perceive and respond to human non-verbal communication such as emotions. Although several approaches have been proposed to recognize human emotions based on facial expressions or speech, relatively limited work has been done to fuse these two, and other, modalities to improve the accuracy and robustness of the emotion recognition system. This paper analyzes the strengths and the limitations of systems based only on facial expressions or acoustic information. It also discusses two approaches used to fuse these two modalities: decision level and feature level integration. Using a database recorded from an actress, four emotions were classified: sadness, anger, happiness, and neutral state. By the use of markers on her face, detailed facial motions were captured with motion capture, in conjunction with simultaneous speech recordings. The results reveal that the system based on facial expression gave better performance than the system based on just acoustic information for the emotions considered. Results also show the complementarily of the two modalities and that when these two modalities are fused, the performance and the robustness of the emotion recognition system improve measurably.
---
paper_title: Emotion Recognition based on Phoneme Classes

paper_content:

Recognizing human emotions/attitudes from speech cues has gained increased attention recently. Most previous work has focused primarily on suprasegmental prosodic features calculated at the utterance level for this purpose. Notably, not much attention is paid to details at the segmental phoneme level in the modeling. Based on the hypothesis that different emotions have varying effects on the properties of the different speech sounds, this paper investigates the usefulness of phoneme-level modeling for the classification of emotional states from speech. Hidden Markov models (HMM) based on short-term spectral features are used for this purpose using data obtained from a recording of an actress’ expressing 4 different emotional states anger, happiness, neutral, and sadness. We designed and compared two sets of HMM classifiers: a generic set of ”emotional speech” HMMs (one for each emotion) a set of broad phoneticclass based HMMs for each emotion type considered. Five broad phonetic classes were used to explore the effect of emotional coloring on different phoneme classes, and it was found that (spectral properties of) vowel sounds were the best indicator to emotions in terms of the classification performance. The experiments also showed that the best performance can be obtained by using phoneme-class classifiers over generic “emotional” HMM classifier and classifiers based on global prosodic features. To see complementary effect of the prosodic and spectral features, two classifiers were combined at the decision level. The improvement was 0.55% in absolute compared with the result from phoneme-class based HMM classifier.
---
paper_title: Class-level spectral features for emotion recognition

paper_content:

The most common approaches to automatic emotion recognition rely on utterance-level prosodic features. Recent studies have shown that utterance-level statistics of segmental spectral features also contain rich information about expressivity and emotion. In our work we introduce a more fine-grained yet robust set of spectral features: statistics of Mel-Frequency Cepstral Coefficients computed over three phoneme type classes of interest - stressed vowels, unstressed vowels and consonants in the utterance. We investigate performance of our features in the task of speaker-independent emotion recognition using two publicly available datasets. Our experimental results clearly indicate that indeed both the richer set of spectral features and the differentiation between phoneme type classes are beneficial for the task. Classification accuracies are consistently higher for our features compared to prosodic or utterance-level spectral features. Combination of our phoneme class features with prosodic features leads to even further improvement. Given the large number of class-level spectral features, we expected feature selection will improve results even further, but none of several selection methods led to clear gains. Further analyses reveal that spectral features computed from consonant regions of the utterance contain more information about emotion than either stressed or unstressed vowel features. We also explore how emotion recognition accuracy depends on utterance length. We show that, while there is no significant dependence for utterance-level prosodic features, accuracy of emotion recognition using class-level spectral features increases with the utterance length.
---
paper_title: Recognizing emotion in speech

paper_content:

The paper explores several statistical pattern recognition techniques to classify utterances according to their emotional content. The authors have recorded a corpus containing emotional speech with over a 1000 utterances from different speakers. They present a new method of extracting prosodic features from speech, based on a smoothing spline approximation of the pitch contour. To make maximal use of the limited amount of training data available, they introduce a novel pattern recognition technique: majority voting of subspace specialists. Using this technique, they obtain classification performance that is close to human performance on the task.
---
paper_title: Techniques and applications of emotion recognition in speech

paper_content:

Affective computing opens a new area of research in computer science with the aim to improve the way how humans and machines interact. Recognition of human emotions by machines is becoming a significant focus in recent research in different disciplines related to information sciences and Human-Computer Interaction (HCI). In particular, emotion recognition in human speech is important, as it is the primary communication tool of humans. This paper gives a brief overview of the current state of the research in this area with the aim to underline different techniques that are being used for detecting emotional states in vocal expressions. Furthermore, approaches for extracting speech features from speech datasets and machine learning methods with special emphasis on classifiers are analysed. In addition to the mentioned techniques, this paper also gives an outline of the areas where emotion recognition could be utilised such as healthcare, psychology, cognitive sciences and marketing.
---
paper_title: Feature Selection for Speech Emotion Recognition in Spanish and Basque: On the Use of Machine Learning to Improve Human-Computer Interaction

paper_content:

Study of emotions in human–computer interaction is a growing research area. This paper shows an attempt to select the most significant features for emotion recognition in spoken Basque and Spanish Languages using different methods for feature selection. RekEmozio database was used as the experimental data set. Several Machine Learning paradigms were used for the emotion classification task. Experiments were executed in three phases, using different sets of features as classification variables in each phase. Moreover, feature subset selection was applied at each phase in order to seek for the most relevant feature subset. The three phases approach was selected to check the validity of the proposed approach. Achieved results show that an instance-based learning algorithm using feature subset selection techniques based on evolutionary algorithms is the best Machine Learning paradigm in automatic emotion recognition, with all different feature sets, obtaining a mean of 80,05% emotion recognition rate in Basque and a 74,82% in Spanish. In order to check the goodness of the proposed process, a greedy searching approach (FSS-Forward) has been applied and a comparison between them is provided. Based on achieved results, a set of most relevant non-speaker dependent features is proposed for both languages and new perspectives are suggested.
---
paper_title: Detection and Analysis of Emotion From Speech Signals

paper_content:

Recognizing emotion from speech has become one the active research themes in speech processing and in applications based on human-computer interaction. This paper conducts an experimental study on recognizing emotions from human speech. The emotions considered for the experiments include neutral, anger, joy and sadness. The distinuishability of emotional features in speech were studied first followed by emotion classification performed on a custom dataset. The classification was performed for different classifiers. One of the main feature attribute considered in the prepared dataset was the peak-to-peak distance obtained from the graphical representation of the speech signals. After performing the classification tests on a dataset formed from 30 different subjects, it was found that for getting better accuracy, one should consider the data collected from one person rather than considering the data from a group of people.
---
paper_title: Comparative analysis of emotion estimation methods based on physiological measurements for real-time applications

paper_content:

In order to improve intelligent Human-Computer Interaction it is important to create a personalized adaptive emotion estimator that is able to learn over time emotional response idiosyncrasies of individual person and thus enhance estimation accuracy. This paper, with the aim of identifying preferable methods for such a concept, presents an experiment-based comparative study of seven feature reduction and seven machine learning methods commonly used for emotion estimation based on physiological signals. The analysis was performed on data obtained in an emotion elicitation experiment involving 14 participants. Specific discrete emotions were targeted with stimuli from the International Affective Picture System database. The experiment was necessary to achieve the uniformity in the various aspects of emotion elicitation, data processing, feature calculation, self-reporting procedures and estimation evaluation, in order to avoid inconsistency problems that arise when results from studies that use different emotion-related databases are mutually compared. The results of the performed experiment indicate that the combination of a multilayer perceptron (MLP) with sequential floating forward selection (SFFS) exhibited the highest accuracy in discrete emotion classification based on physiological features calculated from ECG, respiration, skin conductance and skin temperature. Using leave-one-session-out crossvalidation method, 60.3% accuracy in classification of 5 discrete emotions (sadness, disgust, fear, happiness and neutral) was obtained. In order to identify which methods may be the most suitable for real-time estimator adaptation, execution and learning times of emotion estimators were also comparatively analyzed. Based on this analysis, preferred feature reduction method for real-time estimator adaptation was minimum redundancy - maximum relevance (mRMR), which was the fastest approach in terms of combined execution and learning time, as well as the second best in accuracy, after SFFS. In combination with mRMR, highest accuracies were achieved by k-nearest neighbor (kNN) and MLP with negligible difference (50.33% versus 50.54%); however, mRMR+kNN is preferable option for real-time estimator adaptation due to considerably lower combined execution and learning time of kNN versus MLP.
---
paper_title: The development of the Athens Emotional States Inventory (AESI): collection, validation and automatic processing of emotionally loaded sentences

paper_content:

AbstractObjectives. The development of ecologically valid procedures for collecting reliable and unbiased emotional data towards computer interfaces with social and affective intelligence targeting patients with mental disorders. Methods. Following its development, presented with, the Athens Emotional States Inventory (AESI) proposes the design, recording and validation of an audiovisual database for five emotional states: anger, fear, joy, sadness and neutral. The items of the AESI consist of sentences each having content indicative of the corresponding emotion. Emotional content was assessed through a survey of 40 young participants with a questionnaire following the Latin square design. The emotional sentences that were correctly identified by 85% of the participants were recorded in a soundproof room with microphones and cameras. A preliminary validation of AESI is performed through automatic emotion recognition experiments from speech. Results. The resulting database contains 696 recorded utterances ...
---
paper_title: Comparison of the Efficiency of Time and Frequency Descriptors Based on Different Classification Conceptions

paper_content:

Extraction and detailed analysis of sound files using the MPEG 7 standard descriptors is extensively explored. However, an automatic description of the specific field of sounds of nature still needs an intensive research. This publication presents a comparison of effectiveness of time and frequency descriptors applied in recognition of species of birds by their voices. The results presented here are a continuation of the research/studies on this subject. Three different conceptions of classification - the WEKA system as classical tool, linguistically modelled fuzzy system and artificial neural network were used for testing the descriptors’ effectiveness. The analysed sounds of birds come from 10 different species of birds: Corn Crake, Hawk, Blackbird, Cuckoo, Lesser Whitethroat, Chiffchaff, Eurasian Pygmy Owl, Meadow Pipit, House Sparrow and Firecrest. For the analysis of the physical features of a song, MPEG 7 standard audio descriptors were used.
---
paper_title: Techniques and applications of emotion recognition in speech

paper_content:

Affective computing opens a new area of research in computer science with the aim to improve the way how humans and machines interact. Recognition of human emotions by machines is becoming a significant focus in recent research in different disciplines related to information sciences and Human-Computer Interaction (HCI). In particular, emotion recognition in human speech is important, as it is the primary communication tool of humans. This paper gives a brief overview of the current state of the research in this area with the aim to underline different techniques that are being used for detecting emotional states in vocal expressions. Furthermore, approaches for extracting speech features from speech datasets and machine learning methods with special emphasis on classifiers are analysed. In addition to the mentioned techniques, this paper also gives an outline of the areas where emotion recognition could be utilised such as healthcare, psychology, cognitive sciences and marketing.
---
paper_title: Fuzzy System for the Classification of Sounds of Birds Based on the Audio Descriptors

paper_content:

This paper presents an application of fuzzy systems for the classification of sounds coded by the selected MPEG-7 descriptors. The model of the fuzzy classification system is based on the audio descriptors for a few chosen species of birds: Great Spotted Woodpecker, Greylag, Goldfinch, Chaffinch. The paper proposes two fuzzy models that definitely differ by the description of the input linguistic variables. The results show, that both approaches are effective. However, second one is more flexible in a case of future expanding of the model with next descriptors or species of birds.
---
paper_title: Facial expression and Emotion

paper_content:

Cross-cultural research on facial expression and the developments of methods to measure facial expression are briefly summarized. What has been learned about emotion from this work on the face is then elucidated. Four questions about facial expression and emotion are discussed. What information does an expression typically convey? Can there be emotion without facial expression? Can there be a facial expression of emotion without emotion? How do individuals differ in their facial expressions of emotion?
---
paper_title: The Human Face as a Dynamic Tool for Social Communication

paper_content:

As a highly social species, humans frequently exchange social information to support almost all facets of life. One of the richest and most powerful tools in social communication is the face, from which observers can quickly and easily make a number of inferences — about identity, gender, sex, age, race, ethnicity, sexual orientation, physical health, attractiveness, emotional state, personality traits, pain or physical pleasure, deception, and even social status. With the advent of the digital economy, increasing globalization and cultural integration, understanding precisely which face information supports social communication and which produces misunderstanding is central to the evolving needs of modern society (for example, in the design of socially interactive digital avatars and companion robots). Doing so is challenging, however, because the face can be thought of as comprising a high-dimensional, dynamic information space, and this impacts cognitive science and neuroimaging, and their broader applications in the digital economy. New opportunities to address this challenge are arising from the development of new methods and technologies, coupled with the emergence of a modern scientific culture that embraces cross-disciplinary approaches. Here, we briefly review one such approach that combines state-of-the-art computer graphics, psychophysics and vision science, cultural psychology and social cognition, and highlight the main knowledge advances it has generated. In the light of current developments, we provide a vision of the future directions in the field of human facial communication within and across cultures.
---
paper_title: Perception of emotional expressions in different representations using facial feature points

paper_content:

Facial expression recognition is an enabling technology for affective computing. Many existing facial expression analysis systems rely on automatically tracked facial feature points. Although psychologists have studied emotion perception from manually specified or marker-based point-light displays, no formal study exists on the amount of emotional information conveyed through automatically tracked feature points. We assess the utility of automatically extracted feature points in conveying emotions for posed and naturalistic data and present results from an experiment that compared human raters' judgements of emotional expressions between actual video clips and three automatically generated representations of them. The implications for optimal face representation and creation of realistic animations are discussed.
---
paper_title: Monitoring chronic disease at home using connected devices

paper_content:

The research purpose is to study the impact’s use of connected devices on cardiovascular diseases. In this paper, we will explain our experimental methodology as well as the first outcomes. Three connected objects are being used for this experiment: a heart rate monitor belt, a tensiometer and a smartwatch. This communication intends to explain the methodology and the procedure that are being used to conduct this project. Our main objective is to monitor participants during their daily routine life, to record and to collect data continuously. This health care monitoring is a case of study of system of systems engineering within it we manage interactions between three aspects: humans, environment and sensors. The idea, then, is to study the different relations and correlations existing between variables coming from those aspects. An important part, of this work, takes into consideration the participant’s emotional aspect (stress, happiness, sadness, among) and analyse that using the appropriate artificial intelligence tools. Being able to detect participant’s emotion, categorize it and analyse its impact on cardiovascular disease, is the main goal of this work.
---
paper_title: Multimodal emotion recognition by combining physiological signals and facial expressions: A preliminary study

paper_content:

Lately, multimodal approaches for automatic emotion recognition have gained significant scientific interest. In this paper, emotion recognition by combining physiological signals and facial expressions was studied. Heart rate variability parameters, respiration frequency, and facial expressions were used to classify person's emotions while watching pictures with emotional content. Three classes were used for both valence and arousal. The preliminary results show that, over the proposed channels, detecting arousal seem to be easier compared to valence. While the classification performance of 54.5% was attained with arousal, only 38.0% of the samples were classified correctly in terms of valence. In future, additional modalities as well as feature selection will be utilized to improve the results.
---
paper_title: Emotion recognition based on EEG features in movie clips with channel selection

paper_content:

Emotion plays an important role in human interaction. People can explain their emotions in terms of word, voice intonation, facial expression, and body language. However, brain–computer interface (BCI) systems have not reached the desired level to interpret emotions. Automatic emotion recognition based on BCI systems has been a topic of great research in the last few decades. Electroencephalogram (EEG) signals are one of the most crucial resources for these systems. The main advantage of using EEG signals is that it reflects real emotion and can easily be processed by computer systems. In this study, EEG signals related to positive and negative emotions have been classified with preprocessing of channel selection. Self-Assessment Manikins was used to determine emotional states. We have employed discrete wavelet transform and machine learning techniques such as multilayer perceptron neural network (MLPNN) and k-nearest neighborhood (kNN) algorithm to classify EEG signals. The classifier algorithms were initially used for channel selection. EEG channels for each participant were evaluated separately, and five EEG channels that offered the best classification performance were determined. Thus, final feature vectors were obtained by combining the features of EEG segments belonging to these channels. The final feature vectors with related positive and negative emotions were classified separately using MLPNN and kNN algorithms. The classification performance obtained with both the algorithms are computed and compared. The average overall accuracies were obtained as 77.14 and 72.92% by using MLPNN and kNN, respectively.
---
paper_title: Multimodal emotion recognition by combining physiological signals and facial expressions: A preliminary study

paper_content:

Lately, multimodal approaches for automatic emotion recognition have gained significant scientific interest. In this paper, emotion recognition by combining physiological signals and facial expressions was studied. Heart rate variability parameters, respiration frequency, and facial expressions were used to classify person's emotions while watching pictures with emotional content. Three classes were used for both valence and arousal. The preliminary results show that, over the proposed channels, detecting arousal seem to be easier compared to valence. While the classification performance of 54.5% was attained with arousal, only 38.0% of the samples were classified correctly in terms of valence. In future, additional modalities as well as feature selection will be utilized to improve the results.
---
paper_title: Multimodal fusion framework: A multiresolution approach for emotion classification and recognition from physiological signals

paper_content:

Abstract The purpose of this paper is twofold: (i) to investigate the emotion representation models and find out the possibility of a model with minimum number of continuous dimensions and (ii) to recognize and predict emotion from the measured physiological signals using multiresolution approach. The multimodal physiological signals are: Electroencephalogram (EEG) (32 channels) and peripheral (8 channels: Galvanic skin response (GSR), blood volume pressure, respiration pattern, skin temperature, electromyogram (EMG) and electrooculogram (EOG)) as given in the DEAP database. We have discussed the theories of emotion modeling based on i) basic emotions, ii) cognitive appraisal and physiological response approach and iii) the dimensional approach and proposed a three continuous dimensional representation model for emotions. The clustering experiment on the given valence, arousal and dominance values of various emotions has been done to validate the proposed model. A novel approach for multimodal fusion of information from a large number of channels to classify and predict emotions has also been proposed. Discrete Wavelet Transform, a classical transform for multiresolution analysis of signal has been used in this study. The experiments are performed to classify different emotions from four classifiers. The average accuracies are 81.45%, 74.37%, 57.74% and 75.94% for SVM, MLP, KNN and MMC classifiers respectively. The best accuracy is for ‘Depressing’ with 85.46% using SVM. The 32 EEG channels are considered as independent modes and features from each channel are considered with equal importance. May be some of the channel data are correlated but they may contain supplementary information. In comparison with the results given by others, the high accuracy of 85% with 13 emotions and 32 subjects from our proposed method clearly proves the potential of our multimodal fusion approach.
---
paper_title: Emotion recognition from physiological signals

paper_content:

Emotion recognition is one of the great challenges in human–human and human–computer interaction. Accurate emotion recognition would allow computers to recognize human emotions and therefore react accordingly. In this paper, an approach for emotion recognition based on physiological signals is proposed. Six basic emotions: joy, sadness, fear, disgust, neutrality and amusement are analysed using physiological signals. These emotions are induced through the presentation of International Affecting Picture System (IAPS) pictures to the subjects. The physiological signals of interest in this analysis are: electromyogram signal (EMG), respiratory volume (RV), skin temperature (SKT), skin conductance (SKC), blood volume pulse (BVP) and heart rate (HR). These are selected to extract characteristic parameters, which will be used for classifying the emotions. The SVM (support vector machine) technique is used for classifying these parameters. The experimental results show that the proposed methodology provides in g...
---
paper_title: Reliable emotion recognition system based on dynamic adaptive fusion of forehead biopotentials and physiological signals

paper_content:

A new dynamic fusion method for designing an emotion recognition system is proposed.A weight is assigned to each classifier based on its performance.The performance of the classifiers during the training and testing phases is considered.Static weights in varying contexts such as emotions do not produce acceptable results.Dynamic weighting strategy improves the performance of the system considerably. In this study, we proposed a new adaptive method for fusing multiple emotional modalities to improve the performance of the emotion recognition system. Three-channel forehead biosignals along with peripheral physiological measurements (blood volume pressure, skin conductance, and interbeat intervals) were utilized as emotional modalities. Six basic emotions, i.e., anger, sadness, fear, disgust, happiness, and surprise were elicited by displaying preselected video clips for each of the 25 participants in the experiment; the physiological signals were collected simultaneously. In our multimodal emotion recognition system, recorded signals with the formation of several classification units identified the emotions independently. Then the results were fused using the adaptive weighted linear model to produce the final result. Each classification unit is assigned a weight that is determined dynamically by considering the performance of the units during the testing phase and the training phase results. This dynamic weighting scheme enables the emotion recognition system to adapt itself to each new user. The results showed that the suggested method outperformed conventional fusion of the features and classification units using the majority voting method. In addition, a considerable improvement, compared to the systems that used the static weighting schemes for fusing classification units, was also shown. Using support vector machine (SVM) and k-nearest neighbors (KNN) classifiers, the overall classification accuracies of 84.7% and 80% were obtained in identifying the emotions, respectively. In addition, applying the forehead or physiological signals in the proposed scheme indicates that designing a reliable emotion recognition system is feasible without the need for additional emotional modalities.
---
paper_title: A Hybrid Model for Automatic Emotion Recognition in Suicide Notes

paper_content:

We describe the Open University team's submission to the 2011 i2b2/VA/Cincinnati Medical Natural Language Processing Challenge, Track 2 Shared Task for sentiment analysis in suicide notes. This Shared Task focused on the development of automatic systems that identify, at the sentence level, affective text of 15 specific emotions from suicide notes. We propose a hybrid model that incorporates a number of natural language processing techniques, including lexicon-based keyword spotting, CRF-based emotion cue identification, and machine learning-based emotion classification. The results generated by different techniques are integrated using different vote-based merging strategies. The automated system performed well against the manually-annotated gold standard, and achieved encouraging results with a micro-averaged F-measure score of 61.39% in textual emotion recognition, which was ranked 1st place out of 24 participant teams in this challenge. The results demonstrate that effective emotion recognition by an automated system is possible when a large annotated corpus is available.
---
paper_title: OpenEAR — Introducing the munich open-source emotion and affect recognition toolkit

paper_content:

Various open-source toolkits exist for speech recognition and speech processing. These toolkits have brought a great benefit to the research community, i.e. speeding up research. Yet, no such freely available toolkit exists for automatic affect recognition from speech. We herein introduce a novel open-source affect and emotion recognition engine, which integrates all necessary components in one highly efficient software package. The components include audio recording and audio file reading, state-of-the-art paralinguistic feature extraction and plugable classification modules. In this paper we introduce the engine and extensive baseline results. Pre-trained models for four affect recognition tasks are included in the openEAR distribution. The engine is tailored for multi-threaded, incremental on-line processing of live input in real-time, however it can also be used for batch processing of databases.
---
paper_title: The EU-Emotion Stimulus Set: A validation study.

paper_content:

The EU-Emotion Stimulus Set is a newly developed collection of dynamic multimodal emotion and mental state representations. A total of 20 emotions and mental states are represented through facial expressions, vocal expressions, body gestures and contextual social scenes. This emotion set is portrayed by a multi-ethnic group of child and adult actors. Here we present the validation results, as well as participant ratings of the emotional valence, arousal and intensity of the visual stimuli from this emotion stimulus set. The EU-Emotion Stimulus Set is available for use by the scientific community and the validation data are provided as a supplement available for download.
---
paper_title: Affective and Behavioural Computing: Lessons Learnt from the First Computational Paralinguistics Challenge

paper_content:

Abstract In this article, we review the INTERSPEECH 2013 Computational Paralinguistics ChallengE (ComParE) – the first of its kind– in light of the recent developments in affective and behavioural computing. The impact of the first ComParE instalment is manifold: first, it featured various new recognition tasks including social signals such as laughter and fillers, conflict in dyadic group discussions, and atypical communication due to pervasive developmental disorders, as well as enacted emotion; second, it marked the onset of the ComParE, subsuming all tasks investigated hitherto within the realm of computational paralinguistics; finally, besides providing a unified test-bed under well-defined and strictly comparable conditions, we present the definite feature vector used for computation of the baselines, thus laying the foundation for a successful series of follow-up Challenges. Starting with a review of the preceding INTERSPEECH Challenges, we present the four Sub-Challenges of ComParE 2013. In particular, we provide details of the Challenge databases and a meta-analysis by conducting experiments of logistic regression on single features and evaluating the performances achieved by the participants.
---

Title: Survey on AI-Based Multimodal Methods for Emotion Detection

Section 1: Introduction
Description 1: This section introduces the field of affective computing, its significance, the need for systems sensitive to human emotions, and the motivation for using multimodal approaches for emotion detection.

Section 2: Multimodal Emotion Analysis
Description 2: This section provides an overview of traditional and advanced techniques used in multimodal emotion analysis, the challenges faced, and notable datasets and methods.

Section 3: Emotion in Text
Description 3: This section discusses the process of identifying emotions in text through opinion mining and sentiment analysis, detailing methodologies and challenges associated with text-based emotion detection.

Section 4: Emotion Detection in the Sound
Description 4: This section explains emotion recognition based on audio signals, including prosodic features and speech analysis, and addresses the technical challenges and methodologies involved.

Section 5: Emotion in Image and Video
Description 5: This section focuses on detecting emotions through nonverbal cues such as facial expressions and gestures captured in images and videos, describing various tools and techniques used for analysis.

Section 6: Existing Tools for Automatic Facial Emotion Recognition
Description 6: This section lists and describes some of the notable tools and software packages available for automatic facial emotion recognition.

Section 7: Emotion Detected by the Physiological and Motor Signals
Description 7: This section explores emotion detection through physiological signals and motor data, detailing the relevant signals, methodologies, and accuracy of various detection techniques.

Section 8: Conclusion
Description 8: This section summarizes the survey, underlining the potential and existing limitations of AI-based multimodal emotion detection methods, and suggests directions for future research.