Learning word representations from large available corpora relies on the distributional hypothesis that words present in similar contexts tend to have similar meanings. Recent work has shown that word representations learnt in this manner lack sentiment information which, fortunately, can be leveraged using external knowledge. Our work addresses the question: can affect lexica improve the word representations learnt from a corpus? In this work, we propose techniques to incorporate affect lexica, which capture fine-grained information about a word's psycholinguistic and emotional orientation, into the training process of Word2Vec SkipGram, Word2Vec CBOW and GloVe methods using a joint learning approach. We use affect scores from Warriner's affect lexicon to regularize the vector representations learnt from an unlabelled corpus. Our proposed method outperforms previously proposed methods on standard tasks for word similarity detection, outlier detection and sentiment detection. We also demonstrate the usefulness of our approach for a new task related to the prediction of formality, frustration and politeness in corporate communication. In natural language research, words, sentences and paragraphs are considered in context through vector space representations, rather than as atomic units with no relational information among them. Although n-gram based methods trained on large volumes of data have been found to outperform more complex approaches both on computational cost and accuracy, the techniques do not scale well in cases where the corpus size is limited(for example, for labeled speech or affect corpora with a size of a few millions of words). Recent work has attempted to improve the performance of word distributions for downstream tasks such as sentiment analysis BID27 and knowledge base completion BID16 using lexical knowledge to enrich word embeddings, by performing methods such as regularization or introducing a loss term in the learning objective.Sentiment relationships between words can be considered transitive, where 'good' < 'better' < 'best' implies that 'good' < 'best'. However, word representations based on traditional approaches such as Word2Vec BID19 and GloVe BID23 are agnostic to the associated sentiments, emotions, or more generally affects BID4 . Furthermore, although words such as delighted and disappointed share similar vector representations given their similar contexts, these words are associated with opposite reactions (or sentiments) as well as have a fairly different interpreted meaning. The challenge in using syntactic relational information for sentiment detection, is that sentiment relations are transitive and symmetric (i.e., if 'delighted' is the opposite of 'disappointed', then 'disappointed' is the opposite of 'delighted'.) Ignoring the bipolar nature of words could lead to spurious results, especially in predictive tasks related to synonyms and antonyms and sentiment analysis. On the other hand, incorporating affect-related information would make word distributions homogeneous and suitable for speech and text generation tasks that aim at capturing author or reader reactions. Furthermore, by using a small sentiment lexicon, it is possible to develop an automatic way to rate words based on their vector space representations. This could help reduce the time and cost required to gather word ratings, as well as eliminate the implicit biases that may be introduced in annotations, such as the high correlation between high valence ratings with high arousal reported by BID27 .We present an approach to build affect-enriched word representations. In other words, we enhance word distributions by incorporating reactions and affect dimensions. The output of this work produces word distributions that capture human reactions by modeling the affect information in the words. The affective word representations distinguish between semantically similar words that have varying affective interpretations. Affect is represented as a weighted relational information between two words, following the approach used by existing work. BID27 identify words of opposite polarity by performing signed spectral clustering on pre-trained embeddings. We present an approach to incorporate external affect and reaction signals in the pre-training step, using the hand-annotated affect lexica to learn from. Our experiments are based on using the state-of-the-art Warriner's affect lexicon BID30 as the input. The proposed approach builds on the intuition that relationships between synonyms and antonyms can be characterized using semantic dictionaries and the relationship can then be deterministically captured into the training loss functions.We evaluate the proposed enriched word distributions on standard natural language tasks. We predict formality, frustration and politeness on a labeled dataset and show improved results using the enriched word embeddings. Further, we outperform the state-of-the-art for sentiment prediction on standard datasets. The key contributions of this paper include:• Algorithm to incorporate affect sensors in the cost functions of distributional word representations (including Word2Vec SkipGram, Word2Vec CBOW, and GloVe) during training using semantic and external affect signals.• Establish the utility of affect enriched word-embeddings for linguistic tasks such as Sentiment and Formality prediction in text data. Our method out performs the state-of-the-art with an 20% improvement in accuracy for the outlier detection methods. Detailed results are reported in table 1.• Introduce a workflow to incorporate affective and reaction signals to word representations during pre-training. We show the generalizability of the workflow through experiments on 3 existing embeddings; Word2Vec-CBOW, Word2Vec-SkipGram, and GloVe.Section 2 covers the prior art in both pre-training and post-training approaches for distributional word representations. Section 3 presents the proposed approach and detailed experiments are discussed in section 4. We conclude with a discussion on the learnings and the observations through this process 5. We find reasonable improvements by our proposed approaches in all the task-based evaluations. SkipGram based methods perform poorly in word similarity prediction and outlier detection, but do well on sentiment and affect prediction. This difference in performance on downstream tasks, has been discussed before in BID9 and BID6 , who point out various issues with word similarity based evaluations such as task subjectivity, low inter annotator agreements and low correlations between the performance of word vectors on word similarity and NLP tasks like text classification, parsing and sentiment analysis. Performance differences can also be attributed to corpus size, which are examined in the Appendix section. Table 3 : Performance of proposed approaches on affect prediction task: (a) In terms of Mean Square Error (MSE) values for affect prediction on a labeled email corpus, (b) Comparison with prior work. The baseline model refers to the corpus only approach, with λ = 0. λ is set to 2 for all other approaches: using Valence list(+V), Arousal(+A), Dominance(+D) and average strength(+VAD).(a ) The results suggest that different embeddings perform well for different tasks. In word similarity tasks, the +V model performs well in GloVe setting but the +A model seems to perform the best for CBOW. Similar results are observed in sentiment prediction: for binary sentiment prediction, arousal scores give the best performance with CBOW embeddings but dominance and valence give the best performance with skip-gram and GloVe embeddings respectively. This suggests that the most flexible method could be an ensemble implementation that considers all these inputs before predicting a final class. Also note that given the vocabulary of our ukWaC corpus as 569, 574 words, our affect lexica with 13, 915 words is relatively small. We plan to take this work forward by further analysis in the future. At the least, we expect superior word embeddings with better quality and larger affect lexica. This work proposes methods to incorporate information from an affect lexicon into Word2Vec and GloVe training process. In a nutshell, we first use WordNet to identify word pairs in the affect lexicon which are semantically related. We define the strength of this relationship using available affect scores. Finally, we modify the training objectives to incorporate this information. In order to evaluate our embeddings, we compare them with baseline approaches where the training completely ignores the affect information. Our embeddings show improvements over baselines on not only Word Similarity benchmarks but also on a more complex, Outlier Detection task. We also do this comparison extrinsically and show that our modified embeddings perform better over prior work in predicting sentiment and predicting formality, frustration and politeness in emails. Among models using Valence, Arousal or Dominance score lists, there is no clear winner but overall addition of valence scores does a reasonable job in almost all of the cases.1. Choosing an appropriate value for hyper-parameter λ:In order to choose a suitable value for λ, we take a 100 MB sample of ukWaC corpus. The sample has close to 20 million tokens, with a vocabulary size of 27,978 words, eliminating all the words having the frequency count of less than 20. We choose a smaller corpus for tuning as it is more manageable with respect to space and time resources.We train a Word2Vec SkipGram model on the above 100MB sample and Valence affect lists by using all the λ value from the set (0, 0.5, 1, 2, 10, 100, 1000) one by one.To pick the most suitable value, we compare the results on word similarity task on the Rubenstein-Goodenough(RG) dataset BID26 . The results are given in FIG2 . Since λ = 2.0 performs the best, we fix this value for all our experiments.
Enriching word embeddings with affect information improves their performance on sentiment prediction tasks. Proposes to use affect lexica to improve word embeddings to outperform the standard Word2vec and Glove. This paper proposes integrating information from a semantic resource quantifying the affect of words into a text-based word embedding algorithm to make language models more reflective of semantic and pragmatic phenomena. This paper introduces modifications the word2vec and GloVe loss functions to incorporate affect lexica to facilitate the learning of affect-sensitive word embeddings.