<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 Computing In Context
Social Sciences Track
Lecture 4--topics, trends, and dimensional scaling
Matthew L. Jones
like, with code and stuff
Step1:  IMPORTANT
Step2:  Let's keep using the remarkable narratives available from Documenting the American South (http
Step3:  back to boolean indexing!
Step4:  For now, we'll play with the cool scientists and use the powerful and fast scikit learn package.
Our Zero-ith tool
Step5:  back to vectorizer from scikit learn
Step6:  so  document_term_matrix is a matrix with 294 rows--the documents--and 1650 columns--the vocabulary or terms or features
Step7:  right now stored super efficiently as a sparse matrix
almost all zeros--good for our computers' limited memory
easier for us to see as a dense matrix
Step8:  While this data frame is lovely to look at and useful to think with, it's tough on your computer's memory
Step9:  that is a symmetrical matrix relating each of the texts (rows) to another text (row)
Step10:  HOMEWORK EXERCISE
Step11:  OMG U...G...L...Y!
WHAT THE? This is nonsense
what's the problem?
we just tried to plot a bunch o' features!
we need only the most significant words!
way to do this
<ASSISTANT_TASK:>
Python Code:
%matplotlib inline
import pandas as pd
import matplotlib.pyplot as plt
import textmining_blackboxes as tm
Explanation: Computing In Context
Social Sciences Track
Lecture 4--topics, trends, and dimensional scaling
Matthew L. Jones
like, with code and stuff
End of explanation
#see if package imported correctly
tm.icantbelieve("butter")
Explanation: IMPORTANT: tm is our temporarily helper, not a standard python package!!
download it from my github:
https://github.com/matthewljones/computingincontext
End of explanation
title_info=pd.read_csv('data/na-slave-narratives/data/toc.csv')
#this is the "metadata" of these files--we'll use today
#why does data appear twice in the filename?
title_info
title_info["Date"].str.replace("\-\?", "5")
title_info["Date"].str.replace("[^0-9]", "") #use regular expressions to clean up
title_info["Date"]=title_info["Date"].str.replace("\-\?", "5")
title_info["Date"]=title_info["Date"].str.replace("[^0-9]", "") # what assumptions have I made about the data?
title_info["Date"]=pd.to_datetime(title_info["Date"], coerce=True)
title_info["Date"]<pd.datetime(1800,1,1)
title_info[title_info["Date"]<pd.datetime(1800,1,1)]
Explanation: Let's keep using the remarkable narratives available from Documenting the American South (http://docsouth.unc.edu/docsouthdata/)
Assuming that you are storing your data in a directory in the same place as your iPython notebook.
Put the slave narratives texts within a data directory in the same place as this notebook
End of explanation
#Let's use a brittle thing for reading in a directory of pure txt files.
our_texts=tm.readtextfiles('data/na-slave-narratives/data/texts')
#again, this is not a std python package
#returns a simple list of the document as very long strings
#note if you want the following notebook will work on any directory of text files.
Explanation: back to boolean indexing!
End of explanation
our_texts=tm.data_cleanse(our_texts)
#more necessary when have messy text
#eliminate escaped characters
Explanation: For now, we'll play with the cool scientists and use the powerful and fast scikit learn package.
Our Zero-ith tool: cleaning up the text
I've included a little utility function in tm that takes a list of strings and cleans it up a bit
check out the code on your own time later
End of explanation
from sklearn.feature_extraction.text import TfidfVectorizer
vectorizer=TfidfVectorizer(min_df=0.5, stop_words='english', use_idf=True)
document_term_matrix=vectorizer.fit_transform(our_texts)
# now let's get our vocabulary--the names corresponding to the rows
vocab=vectorizer.get_feature_names()
len(vocab)
document_term_matrix.shape
Explanation: back to vectorizer from scikit learn
End of explanation
vocab[1000:1100]
Explanation: so  document_term_matrix is a matrix with 294 rows--the documents--and 1650 columns--the vocabulary or terms or features
End of explanation
document_term_matrix_dense=document_term_matrix.toarray()
dtmdf=pd.DataFrame(document_term_matrix_dense, columns=vocab)
dtmdf
Explanation: right now stored super efficiently as a sparse matrix
almost all zeros--good for our computers' limited memory
easier for us to see as a dense matrix
End of explanation
#easy to program, but let's use a robust version from sklearn!
from sklearn.metrics.pairwise import cosine_similarity
similarity=cosine_similarity(document_term_matrix)
#Note here that the `cosine_similiary` can take 
#an entire matrix as its argument
#what'd we get?
similarity
similarity.shape
Explanation: While this data frame is lovely to look at and useful to think with, it's tough on your computer's memory
End of explanation
similarity[100]
#this gives the similarity of row 100 to each of the other rows
Explanation: that is a symmetrical matrix relating each of the texts (rows) to another text (row)
End of explanation
term_document_matrix=document_term_matrix.T
# .T is the easy transposition method for a
# matrix in python's matrix packages.
# import a bunch of packages we need
import matplotlib.pyplot as plt
from sklearn.metrics.pairwise import cosine_similarity
from scipy.cluster.hierarchy import ward, dendrogram
#distance is 1-similarity, so:
dist=1-cosine_similarity(term_document_matrix)
# ward is an algorithm for hierarchical clustering
linkage_matrix=ward(dist)
#plot dendogram
f=plt.figure(figsize=(9,9))
R=dendrogram(linkage_matrix, orientation="right", labels=vocab)
plt.tight_layout()
Explanation: HOMEWORK EXERCISE:
for given document find the most similar and give titles from the csv file
you'll see!
supervised vs. unsupervised learning
slides from class omitted
first example of unsupervised learning
hierarchical clustering
This time we're interested in relations among the words not the texts.
In other words, we're interested in the similarities between one column and another--one term and another term
So we'll work with the transposed matrix--the term-document matrix, rather than the document-term matrix.
For a description of hierarchical clustering, look at the example at https://en.wikipedia.org/wiki/Hierarchical_clustering
End of explanation
vectorizer=TfidfVectorizer(min_df=.96, stop_words='english', use_idf=True)
#try a very high min_df
#rerun the model
document_term_matrix=vectorizer.fit_transform(our_texts)
vocab=vectorizer.get_feature_names()
#check the length of the vocab
len(vocab)
#switch again to the term_document_matrix
term_document_matrix=document_term_matrix.T
dist=1-cosine_similarity(term_document_matrix)
linkage_matrix=ward(dist)
#plot dendogram
f=plt.figure(figsize=(9,9))
R=dendrogram(linkage_matrix, orientation="right", labels=vocab)
plt.tight_layout()
Explanation: OMG U...G...L...Y!
WHAT THE? This is nonsense
what's the problem?
we just tried to plot a bunch o' features!
we need only the most significant words!
way to do this: change the min_df parameter in vectorizer
vectorizer=TfidfVectorizer(min_df=0.5, stop_words='english', use_idf=True)
more an art than a science
End of explanation
<END_TASK>