<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 Computing In Context
Social Sciences Track
Lecture 3--text mining for real
Matthew L. Jones
like, with code and stuff
Step1:  IMPORTANT
Step2:  Let's get some text
Let's use the remarkable narratives available from Documenting the American South (http
Step3:  list comprehensions!
most python thing evah!
how many words in each text within our_texts? can you make a list?
Sure, you could do this as a for loop
for text in our texts
Step4:  How to process text
Python Libraries
Python has an embarrasment of riches when it comes to working with texts. Some libraries are higher level with simpler, well thought out defaults, namely pattern and TextBlob. Most general, of long development, and foundational is the Natural Language Tool Kit--NLTK. The ideas we'll learn to today are key--they have slightly different instantiations in the different tools. Not everything is yet in Python 3, alas!!
nltk 
Step5:  Our first tool
Step6:  for the documentation of sklearn's text data functionality, see http
Step7:  so  document_term_matrix is a matrix with 294 rows--the documents--and 1658 columns--the vocabulary or terms or features
Step8:  right now stored super efficiently as a sparse matrix
almost all zeros--good for our computers' limited memory
easier for us to see as a dense matrix
Step9:  While this data frame is lovely to look at and useful to think with, it's tough on your computer's memory
Now we can throw wide variety of mining algorithms at our data!
Similarity and dissimilarity
We reduced our text to a vector of term-weights.
What can we do once we've committed this real violence on the text?
We can measure distance and similarity
I know. Crazy talk.
Right now our text is just a series of numbers, indexed to words. We can treat it like any collection of vectors more or less. 
And the key way to distinguish two vectors is by measuring their distance or computing their similiarity (1-distance).
You already know how, though you may have buried it along with memories of high school.
Many distance metrics to choose from
key one in textual analysis
Step10:  that is a symmetrical matrix relating each of the texts (rows) to another text (row)
Step11:  HOMEWORK EXERCISE
Step12:  OMG U...G...L...Y!
WHAT THE? This is nonsense
what's the problem?
we just tried to plot a bunch o' features!
we need only the most significant words!
way to do this
<ASSISTANT_TASK:>
Python Code:
%matplotlib inline
import pandas as pd
import matplotlib.pyplot as plt
import textmining_blackboxes as tm
Explanation: Computing In Context
Social Sciences Track
Lecture 3--text mining for real
Matthew L. Jones
like, with code and stuff
End of explanation
#see if package imported correctly
tm.icantbelieve("butter")
Explanation: IMPORTANT: tm is our temporarily helper, not a standard python package!!
download it from my github:
https://github.com/matthewljones/computingincontext
End of explanation
title_info=pd.read_csv('data/na-slave-narratives/data/toc.csv')
#this is the "metadata" of these files--we didn't use today
#why does data appear twice?
#Let's use a brittle thing for reading in a directory of pure txt files.
our_texts=tm.readtextfiles('data/na-slave-narratives/data/texts')
#again, this is not a std python package
#returns a simple list of the document as very long strings
#note if you want the following notebook will work on any directory of text files.
len(our_texts)
our_texts[100][:300] # first 300 words of 100th text
Explanation: Let's get some text
Let's use the remarkable narratives available from Documenting the American South (http://docsouth.unc.edu/docsouthdata/)
Assuming that you are storing your data in a directory in the same place as your iPython notebook.
Put the slave narratives texts within a data directory in the same place as this notebook
End of explanation
lengths=[len(text) for text in our_texts]
Explanation: list comprehensions!
most python thing evah!
how many words in each text within our_texts? can you make a list?
Sure, you could do this as a for loop
for text in our texts:
    blah.blah.blah(our_texts) #not real code
or
for i in range(len(our_texts)
But super easy in python
End of explanation
our_texts=tm.data_cleanse(our_texts)
#more necessary when have messy text
#eliminate escaped characters
Explanation: How to process text
Python Libraries
Python has an embarrasment of riches when it comes to working with texts. Some libraries are higher level with simpler, well thought out defaults, namely pattern and TextBlob. Most general, of long development, and foundational is the Natural Language Tool Kit--NLTK. The ideas we'll learn to today are key--they have slightly different instantiations in the different tools. Not everything is yet in Python 3, alas!!
nltk : grandparent of text analysis packages, cross-platform, complex
crucial for moving beyond bag of words: tagging & other grammatical analysis
pattern : higher level and easier to use the nltk but Python 2.7 only. (wah!)
textblob : even higher level range of natural language processing (3.4 but not yet in conda?)
scikit learn (sklearn): toolkit for scientists, faster, better (use for processing/memory intensive stuff) (Our choice!)
Things we might do to clean up text
tokenization
making .split much better 
Examples??
stemming:
converting inflected forms into some normalized forms
e.g. "chefs" --> "chef"
"goes" --> "go"
"children" --> "child"
stopwords
they are the words you don't want to be included:
"from" "to" "a" "they" "she" "he"
If you need to do lots of such things, you'll want to use ntlk, pattern or TextBlob.
For now, we'll play with the cool scientists and use the powerful and fast scikit learn package.
Our Zero-ith tool: cleaning up the text
I've included a little utility function in tm that takes a list of strings and cleans it up a bit
check out the code on your own time later
End of explanation
from sklearn.feature_extraction.text import TfidfVectorizer
vectorizer=TfidfVectorizer(min_df=0.5, stop_words='english', use_idf=True)
document_term_matrix=vectorizer.fit_transform(our_texts)
Explanation: Our first tool: vectorizer from scikit learn
End of explanation
# now let's get our vocabulary--the names corresponding to the rows
# "feature" is the general term in machine learning and data mining 
# we seek to characterize data by picking out features that will enable discovery
vocab=vectorizer.get_feature_names()
len(vocab)
document_term_matrix.shape
Explanation: for the documentation of sklearn's text data functionality, see http://scikit-learn.org/stable/tutorial/text_analytics/working_with_text_data.html
while this works, mini-lecture on crashes
see kernel above. Therein is the secret to eliminating the dreaded *.
End of explanation
vocab[1000:1100]
Explanation: so  document_term_matrix is a matrix with 294 rows--the documents--and 1658 columns--the vocabulary or terms or features
End of explanation
document_term_matrix_dense=document_term_matrix.toarray()
dtmdf=pd.DataFrame(document_term_matrix_dense, columns=vocab)
dtmdf
Explanation: right now stored super efficiently as a sparse matrix
almost all zeros--good for our computers' limited memory
easier for us to see as a dense matrix
End of explanation
#easy to program, but let's use a robust version from sklearn!
from sklearn.metrics.pairwise import cosine_similarity
similarity=cosine_similarity(document_term_matrix)
#Note here that the `cosine_similiary` can take 
#an entire matrix as its argument
#what'd we get?
similarity
similarity.shape
Explanation: While this data frame is lovely to look at and useful to think with, it's tough on your computer's memory
Now we can throw wide variety of mining algorithms at our data!
Similarity and dissimilarity
We reduced our text to a vector of term-weights.
What can we do once we've committed this real violence on the text?
We can measure distance and similarity
I know. Crazy talk.
Right now our text is just a series of numbers, indexed to words. We can treat it like any collection of vectors more or less. 
And the key way to distinguish two vectors is by measuring their distance or computing their similiarity (1-distance).
You already know how, though you may have buried it along with memories of high school.
Many distance metrics to choose from
key one in textual analysis:
cosine similarity
If $\mathbf{a}$ and $\mathbf{b}$ are vectors, then
$\mathbf{a}\cdot\mathbf{b}=\left\|\mathbf{a}\right\|\left\|\mathbf{b}\right\|\cos\theta$
Or
$\text{similarity} = \cos(\theta) = {A \cdot B \over \|A\| \|B\|} = \frac{ \sum\limits_{i=1}^{n}{A_i \times B_i} }{ \sqrt{\sum\limits_{i=1}^{n}{(A_i)^2}} \times \sqrt{\sum\limits_{i=1}^{n}{(B_i)^2}} }$
(h/t wikipedia)
End of explanation
similarity[100]
#this gives the similarity of row 100 to each of the other rows
Explanation: that is a symmetrical matrix relating each of the texts (rows) to another text (row)
End of explanation
term_document_matrix=document_term_matrix.T
# .T is the easy transposition method for a
# matrix in python's matrix packages.
# import a bunch of packages we need
import matplotlib.pyplot as plt
from sklearn.metrics.pairwise import cosine_similarity
from scipy.cluster.hierarchy import ward, dendrogram
#distance is 1-similarity, so:
dist=1-cosine_similarity(term_document_matrix)
# ward is an algorithm for hierarchical clustering
linkage_matrix=ward(dist)
#plot dendogram
f=plt.figure(figsize=(9,9))
R=dendrogram(linkage_matrix, orientation="right", labels=vocab)
plt.tight_layout()
Explanation: HOMEWORK EXERCISE:
for given document find the most similar and give titles from the csv file
you'll see!
supervised vs. unsupervised learning
slides from class omitted
first example of unsupervised learning
hierarchical clustering
This time we're interested in relations among the words not the texts.
In other words, we're interested in the similarities between one column and another--one term and another term
So we'll work with the transposed matrix--the term-document matrix, rather than the document-term matrix.
For a description of hierarchical clustering, look at the example at https://en.wikipedia.org/wiki/Hierarchical_clustering
End of explanation
vectorizer=TfidfVectorizer(min_df=.96, stop_words='english', use_idf=True)
#try a very high min_df
#rerun the model
document_term_matrix=vectorizer.fit_transform(our_texts)
vocab=vectorizer.get_feature_names()
#check the length of the vocab
len(vocab)
#switch again to the term_document_matrix
term_document_matrix=document_term_matrix.T
dist=1-cosine_similarity(term_document_matrix)
linkage_matrix=ward(dist)
#plot dendogram
f=plt.figure(figsize=(9,9))
R=dendrogram(linkage_matrix, orientation="right", labels=vocab)
plt.tight_layout()
Explanation: OMG U...G...L...Y!
WHAT THE? This is nonsense
what's the problem?
we just tried to plot a bunch o' features!
we need only the most significant words!
way to do this: change the min_df parameter in vectorizer
vectorizer=TfidfVectorizer(min_df=0.5, stop_words='english', use_idf=True)
more an art than a science
End of explanation
<END_TASK>