<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 Word Embedding
This lesson is designed to explore features of word embeddings described by Ben Schmidt in his blog post <a href="http
Step1:  Pre-Processing
Word2Vec learns about the relationships among words by observing them in context. We'll need to tokenize the words in our corpus while retaining sentence boundaries. Since novels were imported as single strings, we'll first use <i>sent_tokenize</i> to divide them into sentences, and second, we'll split each sentence into its own list of words.
We'll use nltk's sentence tokenizer
Step2:  Due to memory and time constraints we'll use our quick and dirty tokenizer
Step3:  First get the sentences
Step4:  Now the words
Step5:  We'll double check that we don't have any empty sentences
Step6:  We should now have a list of lists with sentences and words
Step7:  Word2Vec
Word Embeddings
Word2Vec is the most prominent word embedding algorithm. Word embedding generally attempts to identify semantic relationships between words by observing them in context.
Imagine that each word in a novel has its meaning determined by the ones that surround it in a limited window. For example, in Moby Dick's first sentence, “me” is paired on either side by “Call” and “Ishmael.” After observing the windows around every word in the novel (or many novels), the computer will notice a pattern in which “me” falls between similar pairs of words to “her,” “him,” or “them.” Of course, the computer had gone through a similar process over the words “Call” and “Ishmael,” for which “me” is reciprocally part of their contexts.  This chaining of signifiers to one another mirrors some of humanists' most sophisticated interpretative frameworks of language.
The two main flavors of Word2Vec are CBOW (Continuous Bag of Words) and Skip-Gram, which can be distinguished partly by their input and output during training. Skip-Gram takes a word of interest as its input (e.g. "me") and tries to learn how to predict its context words ("Call","Ishmael"). CBOW does the opposite, taking the context words ("Call","Ishmael") as a single input and tries to predict the word of interest ("me").
In general, CBOW is is faster and does well with frequent words, while Skip-Gram potentially represents rare words better.
Word2Vec Features
<ul>
<li>`size`
Step8:  Embeddings
We can return the actual high-dimensional vector by simply indexing the model with the word as the key
Step9:  gensim comes with some handy methods to analyze word relationships. similarity will give us a number from 0-1 based on how similar two words are. If this sounds like cosine similarity for words, you'd be right! It just takes the cosine similarity of these high dimensional vectors
Step10:  We can also find cosine distance between two clusters of word vectors. Each cluster is measured as the mean of its words
Step11:  We can find words that don't belong with doesnt_match. It finds the mean vector of the words in the list, and identifies the furthest away
Step12:  The most famous implementation of this vector math is semantics. What happens if we take
Step13:  Schmidt looked at words associated with male and female pronouns to investigate gender. Let's try take all the female pronouns and subtracting the male pronouns
Step14:  And the opposite
Step15:  How about together (genderless in Schmidt's sense)?
Step16:  Homework
Use the most_similar method to find the tokens nearest to 'car' in our model. Do the same for 'motorcar'.
What characterizes each word in our corpus? Does this make sense?
Vector addition and subtraction can be thought of in terms of analogy. From the example above
Step17:  We need to get the vector from each word, just like above, and add that to a list
Step18:  We can then calculate pairwise the cosine distance
Step19:  We'll use MDS to reduce the dimensions to two
Step20:  Some fancy matplotlib code...
Step21:  What kinds of semantic relationships exist in the diagram above? Are there any words that seem out of place? How do you think they go there?
Saving/Loading Models
We can save the model as a .txt file with the save_word2vec_format method
Step22:  To load up a model, we just ask gensim. Here's a model trained on Eighteenth Century Collections Online corpus (~2500 texts) made available by Ryan Heuser
Step23:  How does this differ from our novels model?
Homework
Heuser's blog post explores an analogy in eighteenth-century thought that Riches are to Virtue what Learning is to Genius. How true is this in the ECCO-trained Word2Vec model? Is it true in the one we trained?
How might we compare word2vec models more generally?
Alternative features for a classification model
This is really cool but what implications does this have for our model of language? Well, word embeddings are simply more precise features of what we've been trying to get at already. That means we can use them in the machine learning models we've been building.
Recall our DTM bag of words classifier
Step24:  So how can we use word embeddings as features? Believe it or not, one of the most effective ways is to simply average each dimension of our embedding across all the words for a given document. Recall our w2v model for novels was trained for 100 dimensions. Creating the features for a specific document would entail first extracting the 100 dimensions for each word, then average each dimension across all words
Step25:  This gives us a set X array with 100 features. We can write a function to do this for us for any given string
Step26:  We can then featurize all of our documents
Step27:  We can fit and score the machine learning modle just as before
Step28:  What about Heuser's model?
Step29:  Cool! But wait, what if we wanted to know why the model was making decisions. If we ask for the most postive coefficients
Step30:  And the negative
Step31:  These are the indices for the important features. But what are these features now?
Note that using our novels w2v model was not as accurate as a BoW tfidf method. That should be expected given movie review language is likely VERY different from our novel corpus. And our novel corpus likely didn't even have entries for a lot of the words used in our movie reviews corpus.
For modern English, most people look for Stanford's GloVe model. This was trained on 6 billion tokens from Wikipedia and Gigaword! Quite a step up from 150 novels. Even the smallest model is a bit large to be working with on our cloud server, but using this model and the code below, you can see it's power
<ASSISTANT_TASK:>
Python Code:
metadata_tb = Table.read_table('../09-Topic-Modeling/data/txtlab_Novel150_English.csv')
fiction_path = '../09-Topic-Modeling/data/txtlab_Novel150_English/'
novel_list = []
# Iterate through filenames in metadata table
for filename in metadata_tb['filename']:
    
    # Read in novel text as single string, make lowercase
    with open(fiction_path + filename, 'r') as file_in:
        novel = file_in.read()
    
    # Add novel text as single string to master list
    novel_list.append(novel)
Explanation: Word Embedding
This lesson is designed to explore features of word embeddings described by Ben Schmidt in his blog post <a href="http://bookworm.benschmidt.org/posts/2015-10-30-rejecting-the-gender-binary.html">"Rejecting the Gender Binary"</a>.
The primary corpus we use consists of the <a href="http://txtlab.org/?p=601">150 English-language novels</a> made available by the .txtLab at McGill. We also look at a <a href="http://ryanheuser.org/word-vectors-1/">Word2Vec model trained on the ECCO-TCP corpus</a> of 2,350 eighteenth-century literary texts made available by Ryan Heuser. (I have shortened the number of terms in the model by half in order to conserve memory.)
For background on Word2Vec's mechanics, I suggest this <a href="https://www.tensorflow.org/versions/r0.8/tutorials/word2vec/index.html">brief tutorial</a> by Google, especially the sections "Motivation," "Skip-Gram Model," and "Visualizing."
We'll read in Andrew Piper's corpus we used in our Topic Modeling notebook:
End of explanation
from nltk.tokenize import sent_tokenize
Explanation: Pre-Processing
Word2Vec learns about the relationships among words by observing them in context. We'll need to tokenize the words in our corpus while retaining sentence boundaries. Since novels were imported as single strings, we'll first use <i>sent_tokenize</i> to divide them into sentences, and second, we'll split each sentence into its own list of words.
We'll use nltk's sentence tokenizer:
End of explanation
def fast_tokenize(text):
    
    # Iterate through text removing punctuation characters
    no_punct = "".join([char for char in text if char not in punctuation])
    
    # Split text over whitespace into list of words
    tokens = no_punct.split()
    
    return tokens
Explanation: Due to memory and time constraints we'll use our quick and dirty tokenizer:
End of explanation
sentences = [sentence for novel in novel_list for sentence in sent_tokenize(novel)]
Explanation: First get the sentences:
End of explanation
words_by_sentence = [fast_tokenize(sentence.lower()) for sentence in sentences]
Explanation: Now the words:
End of explanation
words_by_sentence = [sentence for sentence in words_by_sentence if sentence != []]
Explanation: We'll double check that we don't have any empty sentences:
End of explanation
words_by_sentence[:2]
Explanation: We should now have a list of lists with sentences and words:
End of explanation
model = gensim.models.Word2Vec(words_by_sentence, size=100, window=5, \
                               min_count=5, sg=0, alpha=0.025, iter=5, batch_words=10000)
Explanation: Word2Vec
Word Embeddings
Word2Vec is the most prominent word embedding algorithm. Word embedding generally attempts to identify semantic relationships between words by observing them in context.
Imagine that each word in a novel has its meaning determined by the ones that surround it in a limited window. For example, in Moby Dick's first sentence, “me” is paired on either side by “Call” and “Ishmael.” After observing the windows around every word in the novel (or many novels), the computer will notice a pattern in which “me” falls between similar pairs of words to “her,” “him,” or “them.” Of course, the computer had gone through a similar process over the words “Call” and “Ishmael,” for which “me” is reciprocally part of their contexts.  This chaining of signifiers to one another mirrors some of humanists' most sophisticated interpretative frameworks of language.
The two main flavors of Word2Vec are CBOW (Continuous Bag of Words) and Skip-Gram, which can be distinguished partly by their input and output during training. Skip-Gram takes a word of interest as its input (e.g. "me") and tries to learn how to predict its context words ("Call","Ishmael"). CBOW does the opposite, taking the context words ("Call","Ishmael") as a single input and tries to predict the word of interest ("me").
In general, CBOW is is faster and does well with frequent words, while Skip-Gram potentially represents rare words better.
Word2Vec Features
<ul>
<li>`size`: Number of dimensions for word embedding model</li>
<li>`window`: Number of context words to observe in each direction</li>
<li>`min_count`: Minimum frequency for words included in model</li>
<li>`sg` (Skip-Gram): '0' indicates CBOW model; '1' indicates Skip-Gram</li>
<li>`alpha`: Learning rate (initial); prevents model from over-correcting, enables finer tuning</li>
<li>`iterations`: Number of passes through dataset</li>
<li>`batch_words`: Number of words to sample from data during each pass</li>
</ul>
Note: cell below uses default value for each argument
Training
We've gotten accustomed to training powerful models in Python with one line of code, why stop now?
End of explanation
model['whale']
Explanation: Embeddings
We can return the actual high-dimensional vector by simply indexing the model with the word as the key:
End of explanation
model.similarity('sense','sensibility')
Explanation: gensim comes with some handy methods to analyze word relationships. similarity will give us a number from 0-1 based on how similar two words are. If this sounds like cosine similarity for words, you'd be right! It just takes the cosine similarity of these high dimensional vectors:
End of explanation
model.n_similarity(['sense','sensibility'],['whale','harpoon'])
Explanation: We can also find cosine distance between two clusters of word vectors. Each cluster is measured as the mean of its words:
End of explanation
model.doesnt_match(['pride','prejudice', 'harpoon'])
Explanation: We can find words that don't belong with doesnt_match. It finds the mean vector of the words in the list, and identifies the furthest away:
End of explanation
model.most_similar(positive=['woman', 'king'], negative=['man'])
Explanation: The most famous implementation of this vector math is semantics. What happens if we take:
$$King - Man + Woman = $$
End of explanation
model.most_similar(positive=['she','her','hers','herself'], negative=['he','him','his','himself'])
Explanation: Schmidt looked at words associated with male and female pronouns to investigate gender. Let's try take all the female pronouns and subtracting the male pronouns:
End of explanation
model.most_similar(positive=['he','him','his','himself'], negative=['she','her','hers','herself'])
Explanation: And the opposite:
End of explanation
model.most_similar(positive=['she','her','hers','herself','he','him','his','himself'], topn=50)
Explanation: How about together (genderless in Schmidt's sense)?
End of explanation
her_tokens = [token for token,weight in model.most_similar(positive=['she','her','hers','herself'], \
                                                       negative=['he','him','his','himself'], topn=50)]
Explanation: Homework
Use the most_similar method to find the tokens nearest to 'car' in our model. Do the same for 'motorcar'.
What characterizes each word in our corpus? Does this make sense?
Vector addition and subtraction can be thought of in terms of analogy. From the example above: 'man' is to 'king' as 'woman' is to '???'. Use the most_similar method to find: 'paris' is to 'france' as 'london' is to '???'
What has our model learned about nation-states?
Perform the canonic Word2Vec addition again but leave out a term. Try 'king' - 'man', 'woman' - 'man', 'woman' + 'king'.
What do these indicate semantically?
Visualization
We can use multi-dimensional scaling to visualize this space just like we did with the documents before. But there are a lot of words here, so let's limit it to 50 words from our female gendered subset:
End of explanation
vectors = [model[word] for word in her_tokens]    
Explanation: We need to get the vector from each word, just like above, and add that to a list:
End of explanation
from sklearn.metrics import pairwise
dist_matrix = pairwise.pairwise_distances(vectors, metric='cosine')
Explanation: We can then calculate pairwise the cosine distance:
End of explanation
from sklearn.manifold import MDS
mds = MDS(n_components = 2, dissimilarity='precomputed')
embeddings = mds.fit_transform(dist_matrix)
Explanation: We'll use MDS to reduce the dimensions to two:
End of explanation
_, ax = plt.subplots(figsize=(10,10))
ax.scatter(embeddings[:,0], embeddings[:,1], alpha=0)
for i in range(len(vectors)):
    ax.annotate(her_tokens[i], ((embeddings[i,0], embeddings[i,1])))
Explanation: Some fancy matplotlib code...
End of explanation
model.wv.save_word2vec_format('word2vec.txtalb_Novel150_English.txt')
Explanation: What kinds of semantic relationships exist in the diagram above? Are there any words that seem out of place? How do you think they go there?
Saving/Loading Models
We can save the model as a .txt file with the save_word2vec_format method:
End of explanation
ecco_model = gensim.models.KeyedVectors.load_word2vec_format('data/word2vec.ECCO-TCP.txt')
ecco_model.most_similar(positive=['woman', 'king'], negative=['man'])
ecco_model.most_similar(positive=['she','her','hers','herself'], negative=['he','him','his','himself'])
Explanation: To load up a model, we just ask gensim. Here's a model trained on Eighteenth Century Collections Online corpus (~2500 texts) made available by Ryan Heuser: http://ryanheuser.org/word-vectors-1/
End of explanation
from nltk.corpus import movie_reviews
from sklearn.pipeline import Pipeline
from sklearn.feature_extraction.text import TfidfVectorizer, CountVectorizer, TfidfTransformer
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import cross_val_score, train_test_split
from sklearn.utils import shuffle
reviews = [movie_reviews.raw(fileid) for fileid in movie_reviews.fileids()]
judgements = [movie_reviews.categories(fileid)[0] for fileid in movie_reviews.fileids()]
np.random.seed(0)
X, y = shuffle(reviews, judgements, random_state=0)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=50)
# get tfidf values
tfidf = TfidfVectorizer()
tfidf.fit(X)
X_train_transformed = tfidf.transform(X_train)
X_test_transformed = tfidf.transform(X_test)
# build and test logit
logit_class = LogisticRegression(penalty='l2', C=1000)
logit_model = logit_class.fit(X_train_transformed, y_train)
logit_model.score(X_test_transformed, y_test)
Explanation: How does this differ from our novels model?
Homework
Heuser's blog post explores an analogy in eighteenth-century thought that Riches are to Virtue what Learning is to Genius. How true is this in the ECCO-trained Word2Vec model? Is it true in the one we trained?
How might we compare word2vec models more generally?
Alternative features for a classification model
This is really cool but what implications does this have for our model of language? Well, word embeddings are simply more precise features of what we've been trying to get at already. That means we can use them in the machine learning models we've been building.
Recall our DTM bag of words classifier:
End of explanation
np.mean([model[w] for w in fast_tokenize(X[0]) if w in model], axis=0)
Explanation: So how can we use word embeddings as features? Believe it or not, one of the most effective ways is to simply average each dimension of our embedding across all the words for a given document. Recall our w2v model for novels was trained for 100 dimensions. Creating the features for a specific document would entail first extracting the 100 dimensions for each word, then average each dimension across all words:
End of explanation
def w2v_featurize(document, model):
    return np.mean([model[w] for w in fast_tokenize(document) if w in model], axis=0)
Explanation: This gives us a set X array with 100 features. We can write a function to do this for us for any given string:
End of explanation
X_train_w2v = [w2v_featurize(d, model) for d in X_train]
X_test_w2v = [w2v_featurize(d, model) for d in X_test]
Explanation: We can then featurize all of our documents:
End of explanation
logit_class = LogisticRegression(random_state=0, penalty='l2', C=1000)
logit_model = logit_class.fit(X_train_w2v, y_train)
logit_model.score(X_test_w2v, y_test)
Explanation: We can fit and score the machine learning modle just as before:
End of explanation
X_train_w2v = [w2v_featurize(d, ecco_model) for d in X_train]
X_test_w2v = [w2v_featurize(d, ecco_model) for d in X_test]
logit_class = LogisticRegression(random_state=0, penalty='l2', C=1000)
logit_model = logit_class.fit(X_train_w2v, y_train)
logit_model.score(X_test_w2v, y_test)
Explanation: What about Heuser's model?
End of explanation
np.argsort(logit_model.coef_[0])[-10:]
Explanation: Cool! But wait, what if we wanted to know why the model was making decisions. If we ask for the most postive coefficients:
End of explanation
np.argsort(logit_model.coef_[0])[:10]
Explanation: And the negative:
End of explanation
X_train_transformed
Explanation: These are the indices for the important features. But what are these features now?
Note that using our novels w2v model was not as accurate as a BoW tfidf method. That should be expected given movie review language is likely VERY different from our novel corpus. And our novel corpus likely didn't even have entries for a lot of the words used in our movie reviews corpus.
For modern English, most people look for Stanford's GloVe model. This was trained on 6 billion tokens from Wikipedia and Gigaword! Quite a step up from 150 novels. Even the smallest model is a bit large to be working with on our cloud server, but using this model and the code below, you can see it's power:
```python
os.system('python -m gensim.scripts.glove2word2vec -i glove.6B.100d.txt -o glove.6B.100d.w2v.txt')
glove = gensim.models.KeyedVectors.load_word2vec_format('glove.6B.100d.w2v.txt')
X_train_glove = [w2v_featurize(d, glove) for d in X_train]
X_test_glove = [w2v_featurize(d, glove) for d in X_test]
logit_class = LogisticRegression(random_state=0, penalty='l2', C=1000)
logit_model = logit_class.fit(X_train_glove, y_train)
logit_model.score(X_test_glove, y_test)
.8125
```
While this is not as accurate as our BoW tfidf method, there have been several applications and transformations of word embeddings that have proven to be more accurate than a BoW tfidf on general modern text corpora. And keep in mind, one of the most interesting parts of this is that it only uses 100 dimensions, i.e., we can get ~81% accuracy by reducing a movie review to only 100 different features (our BoW model had over 39000!).
End of explanation
<END_TASK>