<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 Homework 14 (or so)
Step1:  You can explore the files if you'd like, but we're going to get the ones from convote_v1.1/data_stage_one/development_set/. It's a bunch of text files.
Step2:  So great, we have 702 of them. Now let's import them.
Step3:  In class we had the texts variable. For the homework can just do speeches_df['content'] to get the same sort of list of stuff.
Take a look at the contents of the first 5 speeches
Step4:  Doing our analysis
Use the sklearn package and a plain boring CountVectorizer to get a list of all of the tokens used in the speeches. If it won't list them all, that's ok! Make a dataframe with those terms as columns.
Be sure to include English-language stopwords
Step5:  Okay, it's far too big to even look at. Let's try to get a list of features from a new CountVectorizer that only takes the top 100 words.
Step6:  Now let's push all of that into a dataframe with nicely named columns.
Step7:  Everyone seems to start their speeches with "mr chairman" - how many speeches are there total, and many don't mention "chairman" and how many mention neither "mr" nor "chairman"?
Step8:  What is the index of the speech thank is the most thankful, a.k.a. includes the word 'thank' the most times?
Step9:  If I'm searching for China and trade, what are the top 3 speeches to read according to the CountVectoriser?
Step10:  Now what if I'm using a TfidfVectorizer?
Step11:  What's the content of the speeches? Here's a way to get them
Step12:  Now search for something else! Another two terms that might show up. elections and chaos? Whatever you thnik might be interesting.
Step13:  Enough of this garbage, let's cluster
Using a simple counting vectorizer, cluster the documents into eight categories, telling me what the top terms are per category.
Using a term frequency vectorizer, cluster the documents into eight categories, telling me what the top terms are per category.
Using a term frequency inverse document frequency vectorizer, cluster the documents into eight categories, telling me what the top terms are per category.
Step14:  Which one do you think works the best?
Not sure. The last one term frequency inverse I can't get to work. So I am going with number 2.
Harry Potter time
I have a scraped collection of Harry Potter fanfiction at https
Step15:  Term Frequency Vectorizer
Step16:  Simple Counting Vectorizer
<ASSISTANT_TASK:>
Python Code:
# If you'd like to download it through the command line...
!curl -O http://www.cs.cornell.edu/home/llee/data/convote/convote_v1.1.tar.gz
# And then extract it through the command line...
!tar -zxf convote_v1.1.tar.gz
Explanation: Homework 14 (or so): TF-IDF text analysis and clustering
Hooray, we kind of figured out how text analysis works! Some of it is still magic, but at least the TF and IDF parts make a little sense. Kind of. Somewhat.
No, just kidding, we're professionals now.
Investigating the Congressional Record
The Congressional Record is more or less what happened in Congress every single day. Speeches and all that. A good large source of text data, maybe?
Let's pretend it's totally secret but we just got it leaked to us in a data dump, and we need to check it out. It was leaked from this page here.
End of explanation
# glob finds files matching a certain filename pattern
import glob
# Give me all the text files
paths = glob.glob('convote_v1.1/data_stage_one/development_set/*')
paths[:5]
len(paths)
Explanation: You can explore the files if you'd like, but we're going to get the ones from convote_v1.1/data_stage_one/development_set/. It's a bunch of text files.
End of explanation
speeches = []
for path in paths:
    with open(path) as speech_file:
        speech = {
            'pathname': path,
            'filename': path.split('/')[-1],
            'content': speech_file.read()
        }
    speeches.append(speech)
speeches_df = pd.DataFrame(speeches)
speeches_df.head()
Explanation: So great, we have 702 of them. Now let's import them.
End of explanation
All_speeches = speeches_df['content']
First_five_speeches = speeches_df['content'].head(5)
First_five_speeches
Explanation: In class we had the texts variable. For the homework can just do speeches_df['content'] to get the same sort of list of stuff.
Take a look at the contents of the first 5 speeches
End of explanation
count_vectorizer = CountVectorizer(stop_words='english')
speech_tokens = count_vectorizer.fit_transform(All_speeches)
count_vectorizer.get_feature_names()
All_tokens = pd.DataFrame(speech_tokens.toarray(), columns=count_vectorizer.get_feature_names())
#All_tokens
Explanation: Doing our analysis
Use the sklearn package and a plain boring CountVectorizer to get a list of all of the tokens used in the speeches. If it won't list them all, that's ok! Make a dataframe with those terms as columns.
Be sure to include English-language stopwords
End of explanation
count_vectorizer_100 = CountVectorizer(max_features=100, stop_words='english')
speech_tokens_top100 = count_vectorizer_100.fit_transform(speeches_df['content'])
Explanation: Okay, it's far too big to even look at. Let's try to get a list of features from a new CountVectorizer that only takes the top 100 words.
End of explanation
Top_100_tokens = pd.DataFrame(speech_tokens_top100.toarray(), columns=count_vectorizer_100.get_feature_names())
Top_100_tokens.head()
Explanation: Now let's push all of that into a dataframe with nicely named columns.
End of explanation
speeches_df.info()
Top_100_tokens['No_chairman'] = Top_100_tokens['chairman'] == 0
Top_100_tokens[Top_100_tokens['No_chairman'] == True].count().head(1)
Top_100_tokens['no_mr'] = Top_100_tokens['mr'] == 0
Top_100_tokens[Top_100_tokens['no_mr'] == True].count().head(1)
Explanation: Everyone seems to start their speeches with "mr chairman" - how many speeches are there total, and many don't mention "chairman" and how many mention neither "mr" nor "chairman"?
End of explanation
Top_100_tokens['thank'].sort_values(ascending=False).head(1)
Explanation: What is the index of the speech thank is the most thankful, a.k.a. includes the word 'thank' the most times?
End of explanation
Top_100_tokens['china trade'] = Top_100_tokens['china'] + Top_100_tokens['trade']
Top_100_tokens['china trade'].sort_values(ascending=False).head(3)
Explanation: If I'm searching for China and trade, what are the top 3 speeches to read according to the CountVectoriser?
End of explanation
idf_vectorizer = TfidfVectorizer(stop_words='english', use_idf=True)
Top_100_tokens_idf = idf_vectorizer.fit_transform(All_speeches)
idf_df = pd.DataFrame(Top_100_tokens_idf.toarray(), columns=idf_vectorizer.get_feature_names())
idf_df['china trade'] = idf_df['china'] + idf_df['trade']
idf_df['china trade'].sort_values(ascending=False).head(3)
Explanation: Now what if I'm using a TfidfVectorizer?
End of explanation
# index 0 is the first speech, which was the first one imported.
paths[402]
# Pass that into 'cat' using { } which lets you put variables in shell commands
# that way you can pass the path to cat
!cat {paths[577]}
Explanation: What's the content of the speeches? Here's a way to get them:
End of explanation
All_tokens['chaos'] = All_tokens['chaos'].sort_values(ascending=False) >= 1
All_tokens[All_tokens['chaos'] == True].count().head(1)
Explanation: Now search for something else! Another two terms that might show up. elections and chaos? Whatever you thnik might be interesting.
End of explanation
#simple counting vectorizer,
from sklearn.cluster import KMeans
number_of_clusters = 8
km = KMeans(n_clusters=number_of_clusters)
count_vectorizer = CountVectorizer(stop_words='english')
X = count_vectorizer.fit_transform(All_speeches)
km.fit(X)
print("Top terms per cluster:")
order_centroids = km.cluster_centers_.argsort()[:, ::-1]
terms = count_vectorizer.get_feature_names()
for i in range(number_of_clusters):
    top_ten_words = [terms[ind] for ind in order_centroids[i, :5]]
    print("Cluster {}: {}".format(i, ' '.join(top_ten_words)))
# term frequency vectorizer,
vectorizer = TfidfVectorizer(use_idf=True, stop_words='english')
X = vectorizer.fit_transform(All_speeches)
number_of_clusters = 8
km = KMeans(n_clusters=number_of_clusters)
km.fit(X)
print("Top terms per cluster:")
order_centroids = km.cluster_centers_.argsort()[:, ::-1]
terms = count_vectorizer.get_feature_names()
for i in range(number_of_clusters):
    top_ten_words = [terms[ind] for ind in order_centroids[i, :10]]
    print("Cluster {}: {}".format(i, ' '.join(top_ten_words)))
#term frequency inverse document frequency vectorizer
def oh_tokenizer(str_input):
    words = re.sub(r"[^A-Za-z0-9\-]", " ", str_input).lower().split()
    return words
l2_vectorizer = TfidfVectorizer(use_idf=True, stop_words='english', tokenizer=oh_tokenizer) 
X = l2_vectorizer.fit_transform(speeches_df['content'])
l2_df = pd.DataFrame(X.toarray(), columns=l2_vectorizer.get_feature_names())
for i in range(number_of_clusters):
    top_ten_words = [l2_df[ind] for ind in order_centroids[i, :9]]
    print("Cluster {}: {}".format(i, ' '.join(top_ten_words)))
Explanation: Enough of this garbage, let's cluster
Using a simple counting vectorizer, cluster the documents into eight categories, telling me what the top terms are per category.
Using a term frequency vectorizer, cluster the documents into eight categories, telling me what the top terms are per category.
Using a term frequency inverse document frequency vectorizer, cluster the documents into eight categories, telling me what the top terms are per category.
End of explanation
!curl -O https://github.com/ledeprogram/courses/raw/master/algorithms/data/hp.zip
!unzip hp.zip
import glob
paths = glob.glob('hp/*.txt')
paths[:5]
len(paths)
Harry_Potter_fiction = []
for path in paths:
    with open(path) as Harry_file:
        speech = {
            'pathname': path,
            'filename': path.split('/')[-1],
            'content': Harry_file.read()
        }
    Harry_Potter_fiction.append(speech)
Harry_df = pd.DataFrame(Harry_Potter_fiction)
Harry_df.head()
All_of_Harry = Harry_df['content']
All_of_Harry.head()
Explanation: Which one do you think works the best?
Not sure. The last one term frequency inverse I can't get to work. So I am going with number 2.
Harry Potter time
I have a scraped collection of Harry Potter fanfiction at https://github.com/ledeprogram/courses/raw/master/algorithms/data/hp.zip.
I want you to read them in, vectorize them and cluster them. Use this process to find out the two types of Harry Potter fanfiction. What is your hypothesis?
End of explanation
vectorizer = TfidfVectorizer(use_idf=True, stop_words='english')
X = vectorizer.fit_transform(All_of_Harry)
# KMeans clustering is a method of clustering.
from sklearn.cluster import KMeans
number_of_clusters = 2
km = KMeans(n_clusters=number_of_clusters)
km.fit(X)
print("Top terms per cluster:")
order_centroids = km.cluster_centers_.argsort()[:, ::-1]
terms = vectorizer.get_feature_names()
for i in range(number_of_clusters):
    top_ten_words = [terms[ind] for ind in order_centroids[i, :10]]
    print("Cluster {}: {}".format(i, ' '.join(top_ten_words)))
#Cluster 1 is about Lily and James, whoever they are. Wait: His parents.
#Cluster 2 is about Harry and Hermione.
Explanation: Term Frequency Vectorizer
End of explanation
from sklearn.cluster import KMeans
number_of_clusters = 2
km = KMeans(n_clusters=number_of_clusters)
count_vectorizer = CountVectorizer(stop_words='english')
X = count_vectorizer.fit_transform(All_of_Harry)
km.fit(X)
print("Top terms per cluster:")
order_centroids = km.cluster_centers_.argsort()[:, ::-1]
terms = count_vectorizer.get_feature_names()
for i in range(number_of_clusters):
    top_ten_words = [terms[ind] for ind in order_centroids[i, :10]]
    print("Cluster {}: {}".format(i, ' '.join(top_ten_words)))
Explanation: Simple Counting Vectorizer
End of explanation
<END_TASK>