---
abstract: 'This paper explores the use of language models to predict 20 human traits from users’ Facebook status updates. The data was collected by the myPersonality project, and includes user statuses along with their personality, gender, political identification, religion, race, satisfaction with life, IQ, self-disclosure, fair-mindedness, and belief in astrology. A single interpretable model meets state of the art results for well-studied tasks such as predicting gender and personality; and sets the standard on other traits such as IQ, sensational interests, political identity, and satisfaction with life. Additionally, highly weighted words are published for each trait. These lists are valuable for creating hypotheses about human behavior, as well as for understanding what information a model is extracting. Using performance and extracted features we analyze models built on social media. The real world problems we explore include gendered classification bias and Cambridge Analytica’s use of psychographic models.'
author:
- Andrew Cutler
- Brian Kulis
bibliography:
- 'big5.bib'
title: Inferring Human Traits From Facebook Statuses
---

Introduction
============

Facebook’s 2 billion users spend an average of 50 minutes a day on Facebook, Messenger, or Instagram [@stewart2016facebook]. Industry seeks to obtain, model and actualize this mountain of data in a variety of ways. For example, social media can be used to establish creditworthiness [@suncorp; @khandani2010consumer], persuade voters [@cogburn2011networked; @gonzalez2017hacking], or seek cognitive behavioral therapy from a chatbot [@fitzpatrick2017delivering]. Many of these tasks depend on knowing something about the personal life of the user. When determining the risk of default, a creditor may be interested in a debtor’s impulsiveness or strength of support network. A user’s home town could disambiguate a search term. Or—reflecting society’s values—a social media company may be less willing to flag inflammatory language when the speaker is criticizing their own [@AllanHateSpeach].

Social media’s endlessly logged interactions have also been a boon to understanding human behavior. Researchers have used various social networks to model bullying [@cheng2015antisocial], urban mobility [@noulas2012tale], and the interplay of friendship and shared interests [@yang2011like]. Such studies do not have the benefit of a controlled setting where a single variable can be isolated. However, orders of magnitude more observations in participants’ natural habitat offer more fidelity to lived experience [@kosinski2015facebook]. Additionally subjects can be sampled from countries not so singularly Western, Educated, Industrialized, Rich, and Democratic—or WEIRD, in the parlance of Henrick et al [@henrich_heine_norenzayan_2010].

In this paper we show how readily different personality and demographic information can be extracted from Facebook statuses. Our reported performance is useful to learn how traits are related to online behavior. For example, sensational interests as measured by the Sensational Interest Questionnaire (SIQ) have been studied for internal reliability [@egan1999sensational], relationship to physical aggression [@egan2009sensational], and role in intrasexual competition [@weiss2004sensational]. Yet work connecting SIQ with social media use relies on individually labeling sensational interests in statuses and is only predictive among males [@hagger2011social]. Our model performs well for both males and females without hand-labeling statuses. Similarly, other research found no relationship between satisfaction with life (SWL) and status updates [@wang2014can]; we show modest test set performance. Finally, although Facebook Likes have been shown to be highly predictive of many personal traits [@kosinski2013private], language models with good performance on this dataset have been limited to predicting personality, age, and gender [@schwartz2013personality; @farnadi2016computational; @sap2014developing].

The benchmark also helps assess the efficacy of services that explicitly or implicitly rely on inferring these traits. This is valuable to those developing new services as well as to users concerned about privacy. Of particular interest is the role of psychographic models in Cambridge Analytica’s (CA) marketing strategy. From leaked internal communications, in 2014 CA amassed a dataset of Facebook profiles and traits almost identical to those in the myPersonality dataset [@nyt]. The week after CA’s project became public, Facebook’s stock plummeted \$75 billion [@marketwatch]. One factor in that drop was the belief that Facebook had allowed a third party to create a powerful marketing tool that could manipulate elections [@guardianBannon; @nyt]. There are dozens of publications on the myPersonality dataset. However, this is the first to predict SIQ, fair-mindedness, and self-disclosure, which CA discussed in relation to building user models [@nyt]. Besides performance benchmarks, the other major contribution of this paper are the most highly weighted words to predict each trait. The weights also say something about human behavior. The interpretation here is more complex: regression on tens of thousands of features is fraught with over-fitting and colinearity. Despite those problems, in Section \[interpret\] we argue that the weights can still be treated as a data exploration tool similar to clustering. We provide examples of previously studied relationships that are borne out in the word lists, and believe the lists are a useful tool to develop yet unstudied hypotheses.

Highly weighted features are also an important way to analyze models. We argue in section \[CA\] that a militarism predictor CA may have built is accurate, but extracts obvious features. Additionally, by inspecting the features in an Atheist vs. Agnostic classifier we find many gendered words. We demonstrate the bias empirically, then fix the classifier to be more fair. This approach is instructive for interrogating more critical models built on social media data.

This paper includes many contributions that could stand alone. We show that the text of Facebook statuses can predict user SWL and SIQ. We expand the prediction of political identity from a single spectrum (liberal/conservative) to twelve distinct ideologies with varying levels of overlap and popularity. On that task, we establish state of the art performance with a model that also provides informative features for every pairwise political comparison. We recreate models CA may have built, and report their performance and the type of information they extracted. We bring character level deep learning to gender prediction. To our knowledge, we also set the standard for predicting IQ, fair-mindedness, self-disclosure, race, and religion from Facebook statuses. Finally, we propose a novel method to make classification less biased.

Given the broad scope of this paper, some contributions are given less space than they would typically merit. Even so, we believe it is important to report results on many traits in a single paper. This demonstrates the power of a simple model and allows task difficulty and extracted features to be compared across traits without concerns about changing experimental setup.

Background
==========

myPersonality Dataset
---------------------

From 2008 to 2012, over 7 million Facebook users took the myPersonality quiz produced by the psychologist David Stillwell [@kosinski2015facebook]. After answering at least 20 questions, users were scored on the Big Five personality axes: openness, creativity, extraversion, agreeableness, and neuroticism. Over 3 million of those users agreed to give researchers access to their extant Facebook profile and their personality scores. A much smaller subset of users answered additional questionnaires about their interests, Friends’ personality, belief in astrology, and other personal information. The research community has added to the dataset by providing race labels for several hundred thousand users; representing the text of statuses in terms of their Linguistic Inquiry and Word Count (LIWC) statistics [@pennebaker2001linguistic]; and much more. Labels used in this study are listed in Tables \[acc\_cat\] and \[acc\_cont\], along with descriptive statistics. To see all available labels, visit myPersonality.org.

myPersonality.org lists 43 publications that use this data. Most work explores the relationship between personality and easily extractable features such as number of Friends or Likes, geographic location, or user-Like pairs. For example, user-Like pairs are shown to be better predictors of a personality than one’s spouse [@youyou2015computer]. In 2013, Schwartz et al introduced the open vocabulary approach (or bag of words) to personality, gender, and age prediction [@schwartz2013personality]. This significantly outperforms closed-vocabulary approaches such as LIWC that rely on domain knowledge to assign each word to one or more of 69 categories. For an excellent overview of related work, we direct readers to that paper’s introduction [@schwartz2013personality].

Language Models
---------------

### Bag of Words

The majority of our experiments use bag of words (BoW) term frequency-inverse document frequency (tf-idf) preprocessing followed by $\ell_2$ regularized regression. First, the vocabulary is limited to the $k$ most common words in a given training set. Then a matrix of word counts, $N$, is constructed, where $N_{ij}$ refers to how often word $j$ is used by subject $i$. Each row is normalized to sum to one, moved to a log scale, and divided by $d$, the ratio of documents in which each word appears. In more formal notation, each element of the tf-idf matrix is defined by

$$W_{ij} = \frac{1 + \log\Big(\frac{N_{ij}}{\sum_{i=1}^{k}N_{ij}}\Big)}{d_j}.$$

$W$ is then normalized so each row lies on the unit sphere. $W$ can now be used for linear classification or regression with $\ell_2$ regularization on the parameters. This is commonly called Ridge Regression. For binary classification problems, labels are assigned values of $\{-1,1\}$ and a threshold determines predicted label. For categorical data with more than two labels, we train a classifier on each pair of labels. Predicted label is decided by majority vote of the $\frac{c(c-1)}{2}$ classifiers, where $c$ is the number of classes.

### Character-Level Convolutional Neural Network {#CNN}

For gender prediction, we also train a 49 layer character level convolutional neural network (char-CNN) described in [@conneau2017very]. Much like successful computer vision architectures [@krizhevsky2012imagenet], each character is embedded in continuous space and combined with neighbors by many layers of convolutional filters. Unlike BoW models, CNNs preserve the temporal dimension, allowing the use of syntactic information. While a great advantage, and theoretically more similar to human cognition, this requires different preprocessing. During training, all inputs must be the same length along the temporal axis despite the wide variation in total length of users’ statuses. We chose to split users’ concatenated statuses into chunks of no more than 4000 characters, and no less than 1000, as this is enough text for humans to perform gender classification [@nguyen2014gender]. Each chunk contains roughly 800 words. Chunks from the same user are assigned entirely to either the training or test set. Unfortunately, preprocessing differences do not allow for a direct comparison between methods. However, enforcing the same preprocessing for both models would necessarily limit one.

Labels
------

Tables \[acc\_cont\] and \[acc\_cat\] provide statistics of the continuous and categorical data respectively. What follows is a brief description of each label and how it was collected.

### Gender

is the binary label users supplied when setting up their Facebook account. Offering this information was common before 2008, and mandatory from 2008-2014. In 2014, (after the collection of this dataset) Facebook added 56 more gender options but still uses a binary representation to monetize users [@bivens2017gender].

### Race

labels provided in the dataset are inferred from profile pictures using the Faceplusplus.com algorithm which can identify races termed White, Black, and Asian. A noisy measure of visual phenotype is not the gold standard for the study of race, however, our results indicate it is related to social media use.

### Political identity

is limited to the twelve most common responses: IPA, anarchist, centrist, conservative, democrat, doesn’t care, hates politics, independent, liberal, libertarian, republican, and very liberal. These are heterogenous categories from an open-ended question. No work was done to limit labels to political parties (eg. remove “doesn’t care”), disambiguate misspelled or similar responses (eg. combine “anarchy” and “anarchist” or “liberal” and “very liberal”), or limit responses to one country. To produce the word list for Liberals and Conservatives in Table \[words\_rel\_pol\], we combine “liberal”, “very liberal”, and “democrat’’ as well as “conservative”, “very conservative”, and “republican”. The most likely meaning of IPA is the Independence Party of America, which was in its nascence during this survey. The party is most popular among young people disaffected by the two party system, a sentiment reflected by the users who report IPA.

### Religion

categories were limited to the nine most common responses, and similar labels were combined. Three variants of Catholic—“catholic”,“christian-catholic”, and “romancatholic”—were merged to form Catholic. Likewise, Christian refers to “christian”, “christian-baptist” and “christian-evangelical”. The entire list includes: Atheist, Agnostic, Catholic, Christian, Hindu, and None.

### Belief in star sign

is the user’s response to “Horoscopes provide useful information to help guide my decisions?” Options include: Strongly Agree, Slightly Agree, No Opinion, Slightly Disagree, and Strongly Disagree.

### Personality

is determined on five axes—Openness, Conscientiousness, Extroversion, Agreeableness, and Neurotocism—by a survey. Users answer 20-300 questions which are used to score each personality component on a scale of 1-5. There is a large body of research showing that five factor analysis is explanatory for behavior [@digman1990personality], and its measurement is reproducible [@mccrae1987validation]. That work is now adapting to larger datasets collected online [@kosinski2015facebook].

### Sensational Interests

include Militarism, Violent-Occult, Intellectual Recreation, Occult Credulousness, and Wholesome activities. Users can indicate “Great Dislike”, “Slight Dislike”, “No Opinion”, “Slight Interest”, and “Great Interest” for 28 different items including: “Drugs”, “Paganism”, “Philosophy”, “Survivalism”, and “Vampires and Wolves”. Interest levels are calculated by summing responses from relevant items. The full calculation can be found in [@egan1999sensational].

### IQ

is determined by 20 questions that conform to Raven’s Standard Progressive Matrices. The development and validation of these questions is explained in [@IQkosinski] and [@kosinski2014measurement]. Because performance on IQ tests has been rising at roughly 0.3 points a year over the past century and IQ is defined as mean 100, the scoring of a test is properly defined over an age cohort [@flynn1987massive]. These scores do not take age into account and the mean is 114.

### Satisfaction with life, self-disclosure, and fair-mindedness

are assessed by separate questionnaires. SWL is a measure of global well being somewhat robust to short term mood fluctuations [@diener1985satisfaction].

The Interpretation of Feature Weights {#interpret}
=====================================

A common approach to understand traits in social science is to solve

$$X = UT + \epsilon,$$ where $X$ is observations of subjects, $T$ is the traits of subjects, $U$ is a transition matrix, and $\epsilon$ is model error [@khandani2010consumer; @egan1999sensational; @cooke2004demographic; @pecina2013personality; @quilty2009personality; @tett1991personality; @park2015automatic; @cesare2017detection; @kleinberg2016inherent]. Traits are preferred to be orthogonal to promote compactness without sacrificing modeling power. The Big 5 personality model is both criticized and defended on grounds of trait independence, explanatory power, and measureability, which conforms to the linear model above [@john1999big]. Because the traits are defined by language they will not be completely orthogonal. Additionally, observations are not independent. As such, values in $U$ will have dependencies across both rows and columns. Some traits like personality are used to predict other traits or life events [@egan1999sensational; @tett1991personality]. Learning those relationships can be interpreted as informing our beliefs about column dependencies for $U$ when both traits are part of $T$.

In this paper, $X$ is the tf-idf word matrix, $T$ is defined by our labels, and the model weights are some estimate of $U$ we define as $\hat{U}$. Row dependencies in $\hat{U}$ are based on how words function. For example, ‘camp’ and ‘camping’ perform similar roles in a status. Likewise, the relationship between IQ and agreeableness will be embedded in the columns of $\hat{U}$. However, many of the tasks have little training data and the solution is ill-posed. Regularization encourages generalization, but does not provide any guarantees. Further, sometimes $\epsilon$ dominates the model when observations are not very explanatory or the relationship to a trait is not linear. Given these challenges, what confidence can be placed in the estimate $\hat{U}$?

These problems mirror those faced when clustering data. Clustering does not come with guarantees it will yield sensible answers in diverse scenarios [@kleinberg2003impossibility]. However, it is broadly useful when exploring large sets of data [@jain1999data; @shamir20021; @dixon2003classification]. Similarly, $\hat{U}$ can be viewed as a way of ranking features for exploration. A highly ranked observation is not proof it is important. But several highly ranked observations with functional coherence may suggest a hypothesis; particularly when coupled with domain knowledge of row and column dependencies in $U$.

The 55 most highly weighted features for each label are reported in the Appendix. Though the word lists are shown in order of importance, this ranking is not strict. Different regularization, preprocessing, or train/test splits can alter the ordering, especially when there are few examples. Additionally, more common words with lower weights may be used more often in a model’s prediction, but may not appear at the top of a list. One may use $\ell_1$ regularization to obtain an arbitrary small number of non-zero weights [@meinshausen2009lasso]. This encourages weighting common words and provides more stable rankings. We demonstrate that approach with our IQ model in Section \[IQ\].

There are many well-studied phenomena embedded in the $\hat{U}$ produced by our work. For example, Sarah Palin is the only politician indicated in the liberal word list in Table \[words\_rel\_pol\]. Likewise, Nancy Pelosi ranks just below Ronald Reagan among conservative words. This accords with literature on the memorability of negative ads [@lau2007effects], importance of outgroup prejudice for social identity [@huddy2003group; @branscombe1994collective], and biases women face in politics [@schneider2014measuring; @dolan2010impact]. We hope the many word lists in the appendix will be useful to researchers in the development of new hypotheses.

$\hat{U}$ is also useful to understand models built on social media data. Until recently, the models themselves were not very important. However, machine learning can now be used to estimate sensitive traits such criminal recidivism [@kleinberg2016inherent]. Given the literalness with which estimates are often interpreted, it is essential to note that model weights are causal for the predicted label. In Section \[gender\_bias\] we use our understanding of the input features to characterize information the model extracts to predict religion. This dataset also includes demographic labels, which show predicted religion labels are more gendered than the ground truth.

We hope the included word lists (a) highlight unstudied relationships about these traits (b) illustrate what kind of information is extracted from social media by machine learning systems.

Results and Discussion
======================

Experimental Setup
------------------

All BoW experiments employ the same preprocessing. Users must have over 500 words in the sum of all their statuses. 80% of the data is randomly assigned to the training set; the remaining samples constitute the test set. The vocabulary is limited to the 40,000 most common words in each training set. Words must be used by at least 10 users but no more than 60% of users in the training set. The regularization parameter is tuned via efficient leave one out cross validation [@vehtari2015efficient] when $n<10,000$, and $3$-fold cross validation for larger datasets. All BoW models are implemented using the sklearn library [@scikit-learn]. Table \[acc\_cont\] reports the number of samples and explained variance (EV) of the predictions on continuous data. Table \[acc\_cat\] reports the number of classes, ratio of samples in the dominant class, homogeneity, and performance on tasks with categorical data.

  **Label**                   **N**   **EV**
  --------------------------- ------- --------
  **Personality**                     
   Openness                   84451   0.171
   Conscientiousness          84451   0.120
   Extroversion               84451   0.141
   Agreeableness              84451   0.090
   Neuroticism                84451   0.100
  **Sensational Interests**           
   Militarism                 4074    0.165
   Violent-Occult             4074    0.192
   Intellectual Recreation    4074    0.033
   Occult Credulousness       4074    0.144
   Wholesome Activities       4074    0.108
  Satisfaction With Life      2502    0.034
  Self Disclosure             2006    0.092
  Fair-Mindedness             2006    0.064
  IQ                          1807    0.128

  : Prediction Accuracy on Continuous Data[]{data-label="acc_cont"}

Explained Variance (EV) is 1-$\frac{\mathrm{Var}(y-\hat{y})}{\mathrm{Var}(y)}$, where $\hat{y}$ is the predicted label.

  **Label**             **N**    **Classes**   **Mode**   **Homogeneity**   **F1-score**   **Acc**
  --------------------- -------- ------------- ---------- ----------------- -------------- ---------
  Gender                109104   2             0.598      0.519             0.92           0.903
  Race                  22059    3             0.682      0.52              0.74           0.766
  Political identity    19769    12            0.213      0.133             0.33           0.337
  Religious identity    8388     5             0.488      0.318             0.54           0.541
  Belief in Star Sign   7115     5             0.331      0.245             0.32           0.334

  : Prediction Accuracy on Categorical Data[]{data-label="acc_cat"}

Mode is the ratio of the dominant class. Homogeneity is the probability two random samples will be of the same class. The F1-Score is the harmonic mean of precision and recall. For non-binary labels, the precision and recall for each class is weighted by its support.

  **Model**             **Accuracy**
  --------------------- --------------
  Human Majority Vote   0.840
  LIWC                  0.784
  Tri-grams             0.914
  Tri-grams + LIWC      0.916
  BoW (40k Vocab)       0.903
  BoW (500k Vocab)      0.928
  49 layer char-CNN     0.901

  : Gender Prediction[]{data-label="acc_gender"}

Human baseline is the majority vote (n=210) in gender prediction on Twitter data [@nguyen2014gender]. LIWC and Tri-grams are reported in [@schwartz2013personality].

Performance {#accuracies}
-----------

### Gender

Table \[acc\_gender\] compares our gender predictor to several other methods. The BoW model with a vocabulary of 500,000 yields accuracy of 92.8%, 1.4% more accurate than the tri-gram model reported by Schwartz et al [@schwartz2013personality]. Even though the same dataset is used, the comparison is not direct. The tri-gram model seeks to remove the age information from words, has a larger vocabulary, preserves some temporal relationships in the tri-grams, and draws a different train/test split. Moreover, the preprocessing is more restrictive and only includes users with at least 1000 words. Notwithstanding these discrepancies, which may boost or dampen performance, the results are very similar. When the LIWC representation is added to the tri-grams, there is a slight improvement to 91.6% accuracy. Preprocessing is even less similar for the char-CNN described in the Section \[CNN\]. The human baseline of 84.0% consists of volunteer judgments based on 20-40 user tweets as reported by Nguyen et al [@nguyen2014gender]. This is less text than is available to the other models, and from a different social media platform. But, with 210 volunteer guesses per user, it provides a relevant human baseline.

### Personality

After gender, personality is the most studied trait in this paper. Likewise, Schwartz et al achieve the best results to date [@schwartz2013personality]. They report the square root of EV to two significant digits: 0.42, 0.35, 0.38, 0.31, 0.31. In that format, we are just 0.01 beneath the state of the art for openness and agreeableness, 0.01 better for neuroticism, and equivalent for the remaining traits. As with gender, we achieve this with a simpler model.

### Political Identity

Prediction accuracy of 33.7% is a gain of 11.7% over the baseline strategy of always predicting the mode, ‘doesn’t care’. As noted in the experiments section, training samples are weighted inversely to their class representation; therefore, ignoring any class will result in an equal loss. This does not provide the highest classification accuracy. However, we believe when some classes are sparsely populated an MSE optimal classifier that is highly biased toward the mode should not be the standard. For reference, equal sample weights and the same training scheme yield classification accuracy of 36.3% and a weighted f1 score of 31.6%. Five classes—IPA, hates politics, independent, libertarian, and very liberal—have no representation in the test set predictions. The weighted classifier predicts each class at least once.

According to Preotiuc-Pietro et al., all previous research on predicting political ideology from social media text has used binary labels such as liberal vs conservative or Democrat vs Republican. They broaden the classification task to include seven gradations on the liberal to conservative spectrum [@preoctiuc2017beyond]. When predicting ideological tilt from tweets, they achieve a 2.6% boost over baseline (19.6%) with BoW follow by logistic regression. Word2Vec feature embeddings [@mikolov2013distributed] and multi-target learning with some hand-crafted labels yield an 8.0% boost. From classification along grades of a single spectrum, we significantly expand the task to twelve diverse identities with varying levels of representation and ideological overlap while maintaining classification accuracy.

In Table \[pol\_mat\] we report the matrix of highest weighted words for separating users in each pairwise class comparison. As with race, belief in star sign, and religion, we plan on making expanded pairwise lists available online. In Table \[cm\_pol\] we report the confusion matrix. Note that many errors are between similar labels, such as liberal and democrat. Ease of training, strong performance, and representation of minority classes make a majority vote system of shallow pairwise classifiers a good approach for this task.

For binary comparison, by pooling {‘very liberal’,‘liberal’,‘democrat’} and {‘very conservative’,‘conservative’,‘republican’} we achieve 76.4% accuracy; 12.1% above baseline. Table \[words\_rel\_pol\] shows the top 55 liberal and conservative words.

### Religion

Religion seems to be more difficult to glean from statuses than political identity. At 54.1%, accuracy is a modest 5.3% above guessing the mode. The most highly weighted pairwise words are on Table \[mat\_rel\], and Table \[cm\_rel\] shows the confusion matrix. The most highly weighted word to distinguish someone who is agnostic from an atheist is ‘boyfriend’. This led us to look deeper at that pairwise classifier in Section \[gender\_bias\]. Binary labels were constructed by pooling {‘catholic’, ‘christian-catholic’, ‘romancatholic’, ‘christian’, ‘christian-baptist’} and {‘atheist’, ‘agnostic’,‘none’ }. We achieve 78.0% accuracy, 5.2% above baseline. Those words are on table \[words\_rel\_pol\]. To our knowledge, there is no other multi class religion predictor to which our results can be compared.

### IQ {#IQ}

In a genome wide association meta study of 78,308 individuals, 336 single nucleotide polymorphisms were found to explain 2.1-4.8% of the IQ variance among the test population [@sniekers2017genome]. We achieve 12.8% EV with a model trained on less than 2000 users and their statuses. Using $\ell_1$ regularization to limit the vocabulary to the ten most informative words—final, physics; ayaw, family, friend, heart, lmao, nite, strong, ur—still yields 5.6% percent EV. The relative accuracy of such a trivial model that leverages intuitive features is a helpful comparison for any project predicting this important trait. To our knowledge, this is the only work to date that infers IQ from social media.

The selected features are also informative. Words suggesting intelligence—‘final’ and ‘physics’—are parsimonious and singularly academic. Whereas the university experience is sufficient to find users with high IQ, features inversely related to IQ are more focused on disposition. From table \[words\_big5\], agreeableness is implied by ‘family’ and ‘heart’; conscientiousness is implied by ‘family’ and ‘lmao’; and low openness is implied by ‘ur’. Overall, the list can be characterized as prosocial, or at least concerned with social relationships. Predicting low IQ with prosocial features seems to challenge some previous research.

Gottlieb et al observed that learning disabled children were more likely to engage in solitary play [@gottlieb1986sociometric]. Play has also been observed to be more aggressive [@bryan1976come]. More directly related to our task, McConaughy and Ritter showed a positive correlation between the IQ of learning disabled boys and social competence scores; and a negative correlation between IQ and behavior problem scores [@mcconaughy1986social]. For further review of the subject see [@bellanti2000disentangling].

An MSE optimal classifier seeks to generalize information about samples near the average. This can cause bias when classifying minorities, but is instructive when interpreting features. Features should say something about the majority of our sample, those with IQ near the mean. This explains why antisocial behavior among those with extremely low IQ does not preclude prosocial behavior indicating moderately lower IQ. Reflecting the limitations of this type of study, words like ‘family’, ‘friend’, and ‘heart’ could also be caused by differing norms for social media use or many other factors. Prosocial words predicting lower IQ does however suggest interesting future work.

### Sensational Interests

In this study, SIQ is the easiest continuous variable to predict, even with an order of magnitude less training data than personality. The SIQ asks lists 28 discrete interests like ‘black magic’ and ‘the armed forces’. Very similar terms can be recovered from statuses: ‘zombie’, ‘blood’, ‘vampire’; ‘military’, ‘marines’, ‘training’. Personality tests, on the other hand, ask more abstract questions like ‘I shirk my duties’ for conscientiousness. Many of these duties seem to be extracted in Table \[words\_big5\]: ‘studying’, ‘busy’,‘obstacles’. But many more training examples are required for similar performance.

This is the first work to demonstrate an automatic system for predicting SIQ. Previous research relied on manually counting the number of sensational interests in statuses. The count was only correlated with militarism among men; the relationship was negative for women [@hagger2011social].

### Satisfaction With Life

Previous research cast doubt on the relationship between status updates and SWL [@wang2014can]. The number of positive words used on Facebook nationwide in a given day, week, or month, is inversely correlated with the SWL of that time period’s myPersonality participants. The interpretation of that result is that it “challenges the assumption that linguistic analysis of internet messages is related to underlying psychological states.” Here we show that a BoW model accounts for 3.4% of the variance in SWL scores. Moreover, the most important words the model finds are intuitive. Lower SWL is implied by “fucking”, “hate”, “bored”, “interview”, “sick”, “hospital”, “insomnia”, “farmville”, and “video”. The deleterious effects of joblessness, anger, chronic illness, and isolation are well documented. Words positively associated with SWL—“camping”, “imagination”, “epic”, “cleaned”, “success”—make similar sense.

Conversational AI on Facebook Messenger is an efficacious and scalable way to administer cognitive behavioral therapy [@fitzpatrick2017delivering]. Our results show linguistic analysis can shed light on underlying psychological states. This is important to find users that could benefit from such treatment.

### Belief in Star Sign

Compared to political identity, BSS has seven fewer classes and a far more homogeneous distribution. Even so, the BSS classifier performs slightly worse than the politics classifier and roughly on par to the baseline of predicting the mode. Unlike our race, gender, politics and sensational interests, we don’t wear belief in astrology on our sleeve.

Model Selection
---------------

BoW models are somewhat unintuitive. Humans use syntactic information when decoding language, which the model discards. Yet, for many tasks they achieve state of the art performance. We compare our BoW to a character-level CNN on gender prediction, our most data rich problem. A character-level CNN is well suited to large amounts of messy, user generated data. Pooling layers in a CNN allow generalization of words like “gooooooooo” and “gooooooo”, while BoW must learn distinct weights. Surprisingly, the CNN does not outperform the simple BoW as shown in Table \[acc\_gender\].

We found the choice of prediction model is not as important as preprocessing. In initial experiments, Support Vector Machines [@suykens1999least] and logistic regression, and $\ell_2$ regularized regression yielded similar performance, depending on choice of $n$-grams and whether Singular Value Decomposition was used [@golub1970singular]. We implement ridge regression and classification for simplicity.

Inferring human traits from social media is now being done using deep models [@iyyer2014political; @preoctiuc2017beyond]. That may be useful in some cases, but for this project the deep model offered no performance boost or intuition to underlying human behavior. Perhaps a continuous bag of words [@mikolov2013distributed] and recurrent neural network [@felbo2017using] would have done better, but researchers should not consider deep learning essential for this field. Moreover, any performance gains should be weighed against loss of interpretability.

Cambridge Analytica {#CA}
-------------------

With current technology, Facebook statuses are a better predictor of someone’s IQ than the totality of their genetic material [@sniekers2017genome]. When a marketing firm adds such a tool to their arsenal it is natural to be suspicious. Indeed, The Guardian article that broke the CA story was headlined “‘I made Steve Bannon’s psychological warfare tool’: meet the data war whistleblower” [@guardianBannon]. (Steve Bannon is the former chief executive of the Trump presidential campaign.) However, closer inspection of psychographic models casts doubt on their ability to add value to an advertising campaign, even when the predictions are accurate. In this paper we show that militarism is one of the most easily inferred traits. At 16.5% explained variance, it is more predictable than any of the big 5 personality traits except openness, even with just 5% of the training data. SIQ is also a much stronger predictor of aggressive behavior than the Big 5 [@egan2009sensational]. If this trait was actionable for the Trump campaign, it is interesting that the two most highly weighted features are ‘xbox’ and ‘man’. Gaming interest and gender are already available via Facebook’s advertising platform; reaching that demographic does not require an independent model. Additionally, Steve Bannon’s belief in the political power of gamers predates CA’s psychographic model by a decade [@wiredBannon].

Readers are encouraged to view the word lists in the Appendix through the lens of task accuracy on Tables \[acc\_cont\] and \[acc\_cat\]. They may come to the same conclusion as the Trump campaign who, according to CBS News, “never used the psychographic data at the heart of a whistleblower who once worked to help acquire the data’s reporting – principally because it was relatively new and of suspect quality and value.” [@CBSbig5]. Performance results and extracted features allow for more informed discussion; particularly for SIQ, fair-mindedness and self-disclosure on which we report the first accurate prediction model.

There are limitations to this analysis. Our models only use statuses; Likes and network statistics could increase accuracy. Further, other psychographic traits beyond militarism may be politically useful but have no obvious demographic stand-in. Finally, we don’t have access to CA’s exact dataset and instead built our models on the myPersonality dataset.

  -- -------------- -------------- ------------- -----------
                                                 
                    **Agnostic**   **Atheist**   **Total**
     **Agnostic**   36             33            69
     **Atheist**    28             58            86
     **Total**      64             91            
  -- -------------- -------------- ------------- -----------

  : Agnostic vs Atheist Confusion Matrix[]{data-label="cm_biased"}

  -- ------ -------------- ------------- -----------
                                         
            **Agnostic**   **Atheist**   **Total**
     ****   86             21            107
     ****   34             16            50
     ****   120            37            
  -- ------ -------------- ------------- -----------

  : Agnostic vs Atheist Confusion Matrix[]{data-label="cm_biased"}

  -- -------------- -------------- ------------- -----------
                                                 
                    **Agnostic**   **Atheist**   **Total**
     **Agnostic**   40             29            69
     **Atheist**    31             55            86
     **Total**      71             84            
  -- -------------- -------------- ------------- -----------

  : Fair Agnostic vs Atheist Confusion Matrix[]{data-label="cm_fair"}

  -- ------ -------------- ------------- -----------
                                         
            **Agnostic**   **Atheist**   **Total**
     ****   85             22            107
     ****   31             19            50
     ****   116            41            
  -- ------ -------------- ------------- -----------

  : Fair Agnostic vs Atheist Confusion Matrix[]{data-label="cm_fair"}

Gender Bias in Atheist vs Agnostic Classifier {#gender_bias}
---------------------------------------------

Highly weighted atheist words include “fucking”, “bloody”, “maths”, “degrees”, “disease”, “wifey”, and “religion”. Meanwhile, “beautiful”, “santa”, “friggin”, “thank”, “hubby”, “miles”, and “paperwork” imply the user is agnostic. This paints a picture of academic, male, disagreeable and British atheists. Agnostic words are more positive, female, and related to mundane preparation. A more complete list is shown in Table \[words\_rel\_pol\]. What follows is an empirical analysis of our estimator‘s gender bias, a discussion of fairness, and results debiasing the model.

In this dataset, atheists and agnostics are 33.5% and 50.3% female respectively. This is a stronger female preference for agnosticism than random surveys across the United States which report 32% and 38%, respectively [@pew]. Table \[cm\_biased\] shows the confusion matrices for men and women. The ratio of predicted to true agnostics is 0.945 for men and 1.35 for women. Similarly, the ratio of false atheist to false agnostic predictions is 90.8% larger for men than women. The classification of women, the minority in this dataset, is highly distorted.

Models built to generalize information often amplify biases in training data. Cooking videos elicit female pronouns in machine-generated captions 68% more than male pronouns, even though the training shows only 33% more women cooking [@zhao2017men]. Word embeddings used in machine translation [@zou2013bilingual], information retrieval [@clinchant2013aggregating], and student grade prediction [@luo2015predicting] produce analogies such as “man is to computer programmer as woman is to homemaker”[@bolukbasi2016man].

There are many notions of fairness defined over an individual [@dwork2012fairness; @joseph2016rawlsian; @kusner2017counterfactual], population [@zafar2017fairness; @hardt2016equality], or information available to the model [@grgic2016case]. Building a fair estimator often requires domain knowledge to define a similarity metric [@dwork2012fairness], make corpus-level constraints [@zhao2017men], or construct a causal model that separates protected information from other latent variables [@kusner2017counterfactual]. In this paper, we will use the notion of Disparate Mistreatment to measure fairness [@zafar2017fairness]. That is, if protected classes experience disparate rates of false positive, false negative or overall misclassification, the estimator is unfair.

To mitigate Disparate Mistreatment we explicitly encode gender—{$-1$,0,$1$} for {male, unknown, female}—in the feature vector during train time. At test time the gender of all samples is encoded as unknown. The intuition is that latent variables are amplified when they are easy to extract and correlated with the target. As demonstrated by the accuracy of our race and gender predictors, that is often the case for protected information. There often exist more informative, if more subtle, traits than the protected features. For example, atheists and agnostics report a yawning gap in those that don’t believe in God, at 92% and 41% [@pew]. Additionally, religiosity is shown to be correlated with both Agreeableness and Conscientiousness [@saroglou2010religiousness]. But gender is much easier to extract then belief in God or personality. By explicitly giving the model gender information, we hope that the model will do more to extract those other features.

This approach produces much less Disparate Mistreatment of men and women. The ratio of predicted to true agnostics moves closer to parity at 1.02 for men and 1.22 for women. Additionally, the ratio of false atheist to false agnostic predictions is now only 31.8% larger for men, compared to 90.8% without intervention. The most highly weighted agnostic words for the new fair classifier are also less gendered; “hair”, “wifey”, and “boyfriend” are no longer in the top 55, as reported in Table \[words\_rel\_pol\]. We also saw no decay in classification rate.

The gender bias of the atheism classifier is clear by simply inspecting its most heavily weighted features. More opaque models should be subjected to more rigorous inspection for bias.

Conclusion and Future Work
==========================

We match or set the state of the art for the 20 traits in this paper. Additionally, we provide the top words for many pairwise classification problems, and top 55 words for regression or binary classification problems. We hope researchers from many fields find the benchmarks and word lists useful. Our analysis of psychographic models in marketing as well as gender bias in a religion classifier are examples of how these performance measures and extracted features can be used together.

In future work we hope to explore what types of unfairness can be solved by our approach in Section \[gender\_bias\]. Further, models built on traits with few examples are well suited to be augmented by transfer learning. This is especially pressing for detecting states like low satisfaction with life, which can be somewhat ameliorated at low cost.

[angle=270]{}

  -- --------- --------------- -------------- -------------- ----------- ------------------ ---------------- ------------ ---------- ------------ ------------ -------------
     **IPA**   **anarchist**   **centrist**   **conserv.**   **dem.**    **doesn’t care**   **hates pol.**   **indep.**   **lib.**   **liber.**   **repub.**   **v. lib.**
               fuck            wishes         wishes         smh         yay                rain             congrats     wishes     money        church       damn
     excited                   wishes         driving        excited     lol                dont             driving      excited    ready        ready        excited
     xd        fuck                           lord           today       tattoo             shit             surgery      shit       government   school       damn
     xd        fuck            damn                          fb          anymore            shit             damn         damn       art          school       damn
     xd        fuck            wishes         tonight                    stupid             fuck             died         wishes     government   church       wishes
     packers   fuck            wishes         lord           smh                            shit             definitely   wishes     government   church       damn
     class     music           dey            loves          fb          tht                                 movie        wishes     email        camp         damn
     xd        fuck            wishes         lord           valentine   sitting            fuck                          wishes     beer         parents      damn
     xd        fuck            final          lord           im          xd                 im               gonna                   government   church       damn
     xd        fuck            headache       lord           walk        xd                 dont             till         packing                 girls        vacation
     xd        fuck            wishes         wishes         smh         mum                fuck             minute       wishes     fucking                   damn
     xd        xd              boy            lord           im          xd                 xd               school       missing    im           im           
  -- --------- --------------- -------------- -------------- ----------- ------------------ ---------------- ------------ ---------- ------------ ------------ -------------

  : Pairwise Politics Words[]{data-label="pol_mat"}

[angle=270]{}

  -- --------- ----------- -------------- -------------- ---------- ------------------ ---------------- ------------ ---------- ------------ ------------ ------------- -----------
                                                                                                                                                                        
     **IPA**   **anar.**   **centrist**   **conserv.**   **dem.**   **doesn’t care**   **hates pol.**   **indep.**   **lib.**   **liber.**   **repub.**   **v. lib.**   **Total**
     **0**     2           3              3              11         18                 2                1            3          1            16           1             61
     0         **24**      4              3              5          21                 1                3            15         5            4            3             88
     2         9           **74**         40             52         66                 3                6            95         7            43           4             401
     2         5           29             **113**        26         31                 0                7            53         5            62           0             333
     5         17          53             36             **321**    101                4                18           80         9            89           3             736
     3         39          51             29             122        **373**            12               12           105        12           102          9             869
     0         4           6              1              6          30                 **5**            3            6          0            2            0             63
     0         8           16             13             35         22                 1                **8**        29         4            25           1             162
     1         18          51             27             74         51                 6                6            **223**    15           24           13            509
     0         12          17             9              17         28                 0                6            32         **11**       12           4             148
     1         8           19             57             67         64                 1                8            29         3            **179**      3             439
     0         4           25             2              11         22                 2                2            67         1            6            **3**         145
     14        150         348            333            747        827                37               80           737        73           564          44            3954
  -- --------- ----------- -------------- -------------- ---------- ------------------ ---------------- ------------ ---------- ------------ ------------ ------------- -----------

  : Politics Confusion Matrix[]{data-label="cm_pol"}

  -- ------------- -------------- -------------- --------------- ----------
     **athiest**   **agnostic**   **catholic**   **christian**   **none**
                   boyfriend      thank          church          lol
     fucking                      prayers        church          lol
     fucking       fucking                       lol             lol
     fucking       fucking        mass                           xmas
     fucking       apartment      god            church          
  -- ------------- -------------- -------------- --------------- ----------

  : Pairwise Religion Words[]{data-label="mat_rel"}

The most highly weighted word from each pairwise classifier. Word implies top label.

  -- ------------- -------------- -------------- --------------- ---------- -----------
                                                                            
     **Atheist**   **Agnostic**   **Catholic**   **Christian**   **None**   **Total**
     **68**        29             17             16              21         151
     54            **69**         27             55              11         216
     27            37             **172**        130             9          375
     35            48             126            **560**         26         795
     22            11             19             50              **39**     141
     206           194            361            811             106        1678
  -- ------------- -------------- -------------- --------------- ---------- -----------

  : Religion Confusion Matrix[]{data-label="cm_rel"}

In the remaining tables the top 55 words are listed in order for each trait.

\[words\_big5\]

  -------------- ------------ ----------------- --------------- ------------ ------------
                                                                             
  **-**          **+**        **-**             **+**           **-**        **+**
  bored          art          lost              gym             internet     party
  boring         poetry       fucking           ready           quiet        guys
  husband        beautiful    xd                weekend         bored        amazing
  attitude       universe     phone             excited         listening    audition
  shopping       peace        im                success         apparently   baby
  dinner         poem         bored             finished        computer     haha
  tv             writing      fuck              studying        stupid       dance
  game           books        gonna             busy            pc           girls
  proud          theatre      sick              vacation        hmm          fabulous
  ur             dream        procrastination   arm             anime        blast
  dentist        mind         internet          officially      tt           ready
  daughter       book         computer          family          dark         im
  dont           woman        probably          relax           probably     wine
  haha           guitar       cousins           tennis          sims         success
  stupid         damn         hates             wonderful       didn         lets
  ni             awesome      sims              special         watching     excited
  ipod           tea          anybody           win             slow         super
  bed            apartment    charger           glad            depressing   text
  justin         insomnia     sister            piano           calculus     chill
  gift           xd           playing           scholarship     kind         phone
  2nd            adventure    grounded          received        anymore      dear
  hurt           cali         poker             lmao            repost       parties
  ohh            far          tt                degrees         maybe        support
  baseball       philosophy   status            state           draw         loves
  mum            sigh         momma             tons            yay          pics
  pray           nature       ftw               motor           trying       hey
  school         maybe        press             obstacles       books        big
  repost         music        dead              research        shadow       hit
  booked         blues        failed            extremely       bother       met
  lord           chill        forgot            circumstances   damned       pirate
  ops            fam          depression        workout         suppose      ben
  nice           epic         lazy              paid            reading      rocked
  tmr            places       youtube           100             cat          gang
  dam            rights       420               hit             poor         sex
  idol           dragons      school            surgery         depression   sing
  snowing        woot         http              law             sigh         btw
  pissed         vampire      awsome            university      games        gorgeous
  shut           soul         pokemon           anatomy         drawing      musical
  maths          eclipse      woke              blessings       odd          cali
  msn            drawing      dammit            hmmmm           10th         girlfriend
  aldean         strange      hair              husband         pokemon      stoked
  vodka          planet       wished            counting        nice         folks
  comes          yay          cleaning          calc            essay        ponder
  eid            dreams       fine              louis           pointless    wanna
  alot           blood        dunno             delhi           managed      hahahaha
  waste          sushi        enemy             final           looks        pool
  worst          smoking      social            drive           grr          tanning
  kiero          contact      yo                lets            darkness     hello
  soo            lines        procrastinator    iphone          saw          pumped
  mas            deep         black             lunch           crying       chillin
  staff          genius       magic             yankees         lonely       theatre
  12             novel        wasn              running         laptop       kiss
  piss           smh          fans              weather         shouldn      office
  transformers   worried      kinda             zone            paranoid     cock
  car            folks        trying            smart           walking      lauren
  -------------- ------------ ----------------- --------------- ------------ ------------

  : Personality Words

\[words\_big5\_cont\]

  ----------- --------------- -------------- ------------ ------------ -----------
                                                                       
  **-**       **+**           **-**          **+**        **-**        **+**
  fucking     wonderful       loving         sick         bored        family
  stupid      amazing         girlfriend     nervous      fuck         loving
  kill        awesome         wife           stressed     fucking      hope
  shopping    haha            awesome        depression   hates        thankful
  shit        smile           parties        depressed    bday         india
  burn        happiness       party          anymore      apparently   wonderful
  bitch       phone           weekend        lonely       damn         busy
  pissed      urself          haha           stress       internet     friend
  punch       family          doing          fucking      zero         heart
  hates       blessed         game           tired        chem         man
  death       status          sunday         trying       wat          yum
  hell        music           kansas         depressing   supposed     fb
  suck        woop            guy            sims         ma           glad
  freak       hands           delicious      anxiety      hating       beautiful
  piss        heart           beach          worst        spend        lauren
  dead        spirit          definitely     hair         la           lord
  xmas        smiles          swag           fed          dumb         wine
  karma       guy             started        scream       young        swim
  fight       moment          ready          fine         british      energy
  blood       beautiful       hunting        nightmare    killed       lunch
  awful       movie           power          rip          hmm          locked
  deal        theres          funniest       tears        france       woot
  misery      car             melody         horrible     chances      sons
  fuck        dancing         hawaii         flu          simply       special
  enemies     lord            action         worse        exams        trust
  fake        guitar          hit            issues       mum          wish
  pathetic    sore            chillin        scared       main         weeks
  irony       sara            workout        stressful    hate         day
  dumb        help            flow           fml          edge         father
  cunt        walk            portland       care         dnt          tried
  care        excited         seat           shes         party        journey
  devil       prayers         smart          stressing    kept         hospital
  black       knowing         snowboarding   ugh          dat          email
  ich         valentines      knowing        sad          didn         business
  russian     borrow          sore           gary         months       santa
  idiots      laura           greatest       hates        du           walked
  cunts       notifications   success        die          rain         lights
  wtf         beard           basketball     actually     pass         kingdom
  crap        reli            update         scary        bus          work
  truck       snowboarding    gf             boyfriend    okay         lol
  deleted     sorry           women          pills        australia    mommy
  anger       chillin         gotta          crying       shooting     turkey
  die         hill            followed       kitty        england      nap
  tu          whats           jumping        awful        africa       revenge
  nightmare   hearts          fool           hurt         rachel       truly
  annoyed     kindness        dancing        bored        fml          son
  rip         study           greatness      fair         metal        final
  bloody      worry           blast          screaming    uk           reached
  drama       clients         woke           dreading     school       survived
  bitches     smells          ass            friggin      wtf          dont
  stupidity   troops          hitting        suicide      matt         0
  hair        sing            cock           miserable    freakin      god
  wifi        goood           wise           quiet        15           kitchen
  fat         holy            kiss           xd           200          normal
  rage        faster          toes           sadness      free         blessing
  ----------- --------------- -------------- ------------ ------------ -----------

  : Personality Words Continued

\[words\_siq\]

  ------------ ---------- ----------- ----------- ------------- ------------
                                                                
  **-**        **+**      **-**       **+**       **-**         **+**
  sleeping     man        lord        hell        im            life
  ugh          xbox       pray        zombie      course        jon
  sad          gets       cousins     damn        boring        beautiful
  excited      gotta      church      fuck        painful       dancing
  lovely       good       michael     bitch       decision      yoga
  oh           training   allah       ass         hurts         thankful
  hair         headed     jesus       drink       bus           peace
  shopping     truck      game        blood       game          kinda
  husband      guitar     0           lmao        stupid        truly
  sick         guys       summer      xd          bak           la
  cares        bro        gosh        woot        hero          ich
  mum          gun        praise      halloween   problem       miss
  boyfriend    boom       sunday      play        yeah          likes
  lady         epic       dad         guys        christ        comfort
  concert      work       loving      drunk       gona          lol
  today        weight     mum         thanx       id            wtf
  gaga         gym        team        animal      sittin        insomnia
  okay         bike       hospital    sanity      die           chicken
  pic          dang       10          fucking     horse         children
  adorable     game       tv          dragons     yell          tired
  sunday       blast      christ      burn        chuck         lovely
  ordered      lol        heal        vampires    2day          ap
  birth        war        usa         blah        tommorrow     funny
  lots         black      personal    man         ow            things
  poor         fish       best        loved       bored         man
  ben          military   ray         pissed      fukin         simple
  fine         woot       nervous     lil         inbox         thank
  settings     12         thing       bday        race          period
  birthday     till       look        send        basketball    countdown
  cousins      ppl        week        body        word          baby
  shoes        brave      2morrow     metal       rhys          beach
  art          17         quite       head        tell          hey
  omg          fight      poor        piss        step          depression
  stop         success    brazil      blast       wats          jobs
  wear         marines    cup         theyre      coke          cure
  prince       hrs        zumba       cause       football      manage
  round        sword      account     gun         penguins      sugar
  come         make       website     death       won           aware
  neighbours   ko         tryna       vampire     facebookers   singing
  basement     friend     study       bleh        letters       egg
  music        hit        haha        tattoo      awsome        taste
  speak        play       soccer      ppl         dont          rains
  thoughts     pics       feeling     dead        blah          log
  story        hahaha     christmas   woman       till          taught
  weird        troops     round       purple      playing       coolest
  awful        army       youth       peaceful    dead          yellow
  quite        running    story       message     fact          cheers
  rachel       mag        bible       shit        learned       small
  hear         strong     woah        angel       visit         society
  alice        knw        grace       kinda       address       fly
  tea          beer       prayers     tongue      14            social
  promised     hehehe     plan        sushi       chilling      boo
  jesus        comwatch   feat        wolf        win           beauty
  actually     xoxo       anybody     poke        pokemon       world
  counting     run        stressed    kick        sees          sunshine
  ------------ ---------- ----------- ----------- ------------- ------------

  : Sensational Interest Words

\[words\_siq\_cont\]

  ------------- ----------- ----------- ----------- ------------ ------------
                                                                 
  **-**         **+**       **-**       **+**       **No**       **Yes**
  church        zombie      coke        woot        minutes      omg
  praise        ass         michigan    camping     didn         im
  jesus         bitch       stupid      fish        church       ready
  lord          halloween   pathetic    life        praise       friend
  bible         animal      ops         yesterday   jesus        mind
  christ        sign        husband     beautiful   probably     ass
  team          omg         didn        rain        physics      butt
  quite         xd          hurts       man         jess         stay
  loving        job         kurwa       mexico      white        tom
  pray          woot        evil        wish        religion     tomarrow
  paper         wish        afternoon   river       iv           october
  game          cure        problem     love        officially   promise
  blessed       street      taylor      path        imagine      lol
  salvation     vampire     idea        moon        christ       searching
  ops           guys        jess        haha        germany      bitch
  summer        send        glee        snow        giants       bleh
  michael       lol         mum         bike        saw          eye
  spent         thanx       mental      hahaha      wants        cute
  youth         luck        meg         ghost       north        family
  cousins       wtf         mad         baking      decided      halloween
  word          nature      360         grandma     discovered   hanging
  god           cancer      pissed      live        11th         haunted
  homework      woohoo      club        goin        ouch         japanese
  alarm         miss        uni         sky         skin         mother
  0             barely      lyrics      cat         doesn        dinner
  haha          moment      head        animal      bacon        card
  player        bar         recently    netflix     train        help
  sunday        safe        internet    birds       hahaha       bored
  college       proud       min         smile       lasts        luv
  wedding       woman       lesson      happiness   america      luck
  prayer        mom         bus         mom         haven        neighbors
  glory         away        rly         yum         burning      yum
  forgiveness   dare        debate      fishing     pray         fireworks
  ann           inches      kevin       truly       thursday     lmao
  mm            boyfriend   inbox       fell        jessica      tt
  political     il          jeez        make        prince       tired
  fact          nd          official    clean       knew         person
  greatest      pls         nite        portland    umm          nd
  confused      aware       ms          smells      quiero       watch
  appreciated   xmas        lack        lake        deserves     ya
  algebra       hell        saw         create      heres        prom
  brazil        solstice    troy        making      finds        crazy
  travel        date        sims        2010        kim          upload
  daughter      vampires    school      josh        heard        elf
  bacon         copy        thinks      children    punch        hehe
  laura         purple      thanking    laughing    groups       crack
  personal      haunted     die         sa          car          bell
  week          theyre      hates       law         amazing      human
  greater       lmao        stuff       jobs        sick         finish
  statement     later       band        earth       tape         lnk
  messed        interview   thieves     gets        drink        june
  tv            peeps       feels       hehehe      morn         change
  em            peaceful    elm         swimming    dallas       costume
  poor          drunk       germany     wa          cops         shit
  trust         dunno       sat         monkeys     waters       decorating
  ------------- ----------- ----------- ----------- ------------ ------------

  : Sensational Interest Words Continued

\[words\_psych\]

  ------------ ----------- ----------- ------------ ------------- -----------
                                                                  
  **-**        **+**       **-**       **+**        **-**         **+**
  bored        family      bored       excited      nite          exam
  fuck         loving      wat         business     ur            hours
  fucking      hope        soon        says         lmao          sigh
  hates        thankful    dad         apartment    alot          camping
  bday         india       xd          great        family        finish
  apparently   wonderful   stage       delicious    omg           paper
  damn         busy        pass        sure         2011          wtf
  internet     friend      moon        needed       city          il
  zero         heart       haha        seattle      lol           finds
  chem         man         kitty       uni          help          important
  wat          yum         tired       airport      wew           read
  supposed     fb          mum         thankful     boy           physics
  ma           glad        farmville   dallas       heart         google
  hating       beautiful   face        learn        com           ra
  spend        lauren      drank       weekend      angie         xd
  la           lord        fuk         definitely   www           wifi
  dumb         wine        fuck        dinner       ha            text
  young        swim        ma          card         333           weeks
  british      energy      sun         amazing      tom           studying
  killed       lunch       crap        tonight      goodnight     training
  hmm          locked      bday        exciting     history       course
  france       woot        shit        degrees      xxx           student
  chances      sons        hopefully   classes      xdd           magic
  simply       special     feel        support      friend        kinda
  exams        trust       fails       priceless    morning       everytime
  mum          wish        va          oh           mum           raining
  main         weeks       big         certainly    christmas     yea
  hate         day         nd          government   eid           maths
  edge         father      smoke       ticket       kay           semester
  dnt          tried       yay         food         gives         maybe
  party        journey     watchin     january      din           exciting
  kept         hospital    sick        couple       beautiful     point
  dat          email       wedding     php          folks         kno
  didn         business    regret      journey      luv           excited
  months       santa       seconds     universe     0             imma
  du           walked      im          21           hacked        months
  rain         lights      ignore      grateful     secrets       flying
  pass         kingdom     tt          pay          iam           final
  bus          work        lose        size         forgiveness   nah
  okay         lol         marriage    class        strong        library
  australia    mommy       lolz        situation    busy          used
  shooting     turkey      fukin       duke         jo            chem
  england      nap         picture     honesty      hate          brain
  africa       revenge     blessing    austin       ti            everybody
  rachel       truly       slow        tires        nightmare     awesome
  fml          son         anxiety     29           ayaw          groups
  metal        final       cy3         sisters      prayer        progress
  uk           reached     library     mother       fought        champion
  school       survived    tmr         heading      ow            calculus
  wtf          dont        fucking     bc           sana          behave
  matt         0           epic        piece        tired         den
  freakin      god         il          summer       afraid        badly
  15           kitchen     marie       breakfast    para          times
  200          normal      bunch       answer       sum           mobil
  free         blessing    loaded      surgery      movie         fun
  ------------ ----------- ----------- ------------ ------------- -----------

  : Psychographic Words

\[words\_rel\_pol\]

  --------------- ----------- ----------- ------------ ----------- ----------- ------------ --------------
  extra           physics     miles       fucking      church      fucking     church       damn
  miles           fucking     working     physics      pray        fuck        truck        happy
  turn            snowing     extra       wat          prayers     xmas        government   fb
  **hair**        shit        awhile      fuck         god         damn        america      smh
  packing         wat         packing     bloody       easter      shit        pray         marriage
  awhile          write       turn        shit         lord        bloody      haha         xmas
  insane          bloody      super       write        blessed     hell        prayers      chicago
  working         enter       **hubby**   maths        christmas   ass         deer         sex
  **hubby**       fuck        chill       xx           ugh         india       christmas    hell
  points          sigh        free        snowing      praying     zombie      country      fam
  friggin         thinks      sleepy      enter        hw          fuckin      tonight      lovely
  santa           talk        santa       thinks       ppl         halloween   17           halloween
  heck            weeks       heck        talk         prayer      car         lord         health
  wishes          town        ready       science      game        yay         awesome      saw
  child           science     friggin     sigh         believe     social      god          yoga
  free            maths       vacation    hai          family      xx          military     celebrate
  **boyfriend**   degrees     work        cancer       ready       quite       texas        gay
  lady            lolz        thursday    person       fb          religion    freedom      apartment
  learn           record      late        coursework   bless       drink       savior       wtf
  super           xmas        points      town         im          oh          dad          thoughts
  houston         tom         pack        xd           calling     using       bible        shit
  service         hai         houston     weeks        dang        shitty      jesus        glee
  pack            person      insane      tom          paper       internet    supper       gaga
  late            dat         ya          film         jesus       fucked      girls        da
  wanting         tyler       relax       dat          school      damned      huge         palin
  hasn            cod         join        kill         camp        omfg        praying      2010
  mai             afraid      busy        lolz         gosh        meh         camp         help
  sleepy          untill      learn       msn          heart       indian      soldiers     mexico
  worked          present     child       english      success     post        byu          mother
  fly             **wifey**   headed      xmas         mary        head        christ       indian
  chill           movie       favorite    chemistry    strength    cricket     disney       lady
  join            xx          beautiful   afraid       butt        any1        risen        studies
  kyle            cancer      season      na           fishing     dragon      beach        social
  dun             boring      san         pierced      brother     lovely      tournament   art
  thursday        rape        fly         dick         military    body        troops       holiday
  taken           month       worked      anatomy      sad         new         schools      shitty
  childhood       kill        service     bbc          uncle       boyfriend   leave        ve
  mother          welcome     spring      tell         senior      teeth       ill          free
  thank           clinton     wanting     untill       fair        nice        blonde       earthquake
  headed          nicht       halloween   memory       mom         fml         armed        street
  ya              ay          lady        bothered     tan         warped      xbox         phone
  london          brother     thank       horse        watching    woke        reagan       lakers
  beautiful       tell        childhood   record       em          bleh        utah         ur
  jail            hadn        mai         cod          president   wednesday   served       fine
  hates           pierced     hair        ki           smh         gods        tide         relationship
  paperwork       wild        paperwork   nicht        love        afford      gators       asshole
  wanna           use         4th         sheep        haha        japanese    pelosi       worried
  clear           perfect     hopefully   chem         future      tongue      husband      purple
  san             return      missed      brother      best        robert      stinks       putting
  til             needed      peace       fancy        emails      sophie      trial        omg
  halloween       paid        hasn        degrees      goin        holy        picked       nature
  bring           half        trip        disease      football    eye         beep         prop
  kindle          horse       mother      realised     latest      tattoo      gun          black
  vida            disease     sunshine    room         thank       decent      trailer      live
  powers          chuck       kyle        religion     matthew     odd         ready        eid
  --------------- ----------- ----------- ------------ ----------- ----------- ------------ --------------

  : Religion and Politics Words

\[words\_race\]

  -------------- ---------- ------------- ------------- ----------- -------------
  tonight        smh        tonight       asian         smh         korea
  dad            fb         blonde        tt            fb          sa
  stupid         lord       town          tmr           lord        na
  exited         fam        fuckin        korea         wit         asian
  thinks         nigeria    ass           chinese       aint        gay
  ends           yall       college       ng            da          chinese
  journey        black      gas           na            yall        internet
  meet           fathers    dope          korean        lol         korean
  hahahahahaha   mj         worse         china         say         monday
  fun            yuh        night         ang           fam         xd
  awesome        gon        men           aq            jackson     tmr
  ability        birthday   sons          asians        cos         shooting
  night          mad        adult         chen          michael     philippines
  mas            lol        pretty        guys          finals      3d
  wouldnt        finish     theres        thailand      ass         babe
  chargers       dey        idea          taiwan        yuh         heaven
  bein           asap       hope          karaoke       black       important
  aftr           tryna      ability       sa            ny          tan
  pretty         jackson    melissa       chan          sooooo      thailand
  eh             came       state         dream         mad         yummy
  tom            degrassi   unique        company       mind        completely
  exhausted      wat        weekend       craving       season      woot
  tough          iz         screaming     zzz           wat         smell
  great          hw         mamaya        holiday       birthday    bought
  running        pple       tune          wanna         degrassi    fly
  exciting       jus        figure        ms            hell        tt
  yankees        braids     inside        nguyen        chelsea     worry
  politics       haters     exited        singapore     woman       ruin
  mirror         females    wine          yang          figure      passed
  pepsi          misfits    5th           hu            african     skating
  roll           god        superman      fat           nigeria     english
  animal         man        emotionally   ftw           episode     belong
  grr            omg        sell          gg            iz          shot
  gay            african    sitting       rice          smart       mas
  tattoo         desires    february      tttt          saying      grandpa
  2nite          chelsea    easter        damnit        asap        lazy
  spend          female     months        555           attention   sacrifice
  monday         cousin     saying        wong          knowing     grr
  sorrow         holla      expecting     achieve       ki          broken
  ed             smart      rollin        pa            meeting     yang
  healthy        laker      wheres        mode          hw          beer
  enjoyable      favour     eminem        lmao          sings       chatting
  actually       dis        apparently    pride         india       meet
  charity        money      does          bbq           gas         shoulder
  delete         happy      status        super         self        ang
  iron           mii        legit         1st           ready       funn
  blonde         aye        30            long          college     shoes
  comforted      hard       wen           skating       mj          wood
  standards      wuz        eric          mean          search      dad
  shot           ready      yelled        heart         years       apart
  chose          nigga      mis           dx            misfits     aj
  chatting       jamaica    breaking      faith         blessed     line
  damage         bus        homework      expectation   advice      jack
  innocent       facebook   actually      research      boys        totally
  thnx           cos        wishes        hard          fathers     tomorrow
  -------------- ---------- ------------- ------------- ----------- -------------

  : Race Words
