<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 Homework 14 (or so)
Step1:  You can explore the files if you'd like, but we're going to get the ones from convote_v1.1/data_stage_one/development_set/. It's a bunch of text files.
Step2:  So great, we have 702 of them. Now let's import them.
Step3:  In class we had the texts variable. For the homework can just do speeches_df['content'] to get the same sort of list of stuff.
Take a look at the contents of the first 5 speeches
Step4:  Doing our analysis
Use the sklearn package and a plain boring CountVectorizer to get a list of all of the tokens used in the speeches. If it won't list them all, that's ok! Make a dataframe with those terms as columns.
Be sure to include English-language stopwords
Step5:  Okay, it's far too big to even look at. Let's try to get a list of features from a new CountVectorizer that only takes the top 100 words.
Step6:  Now let's push all of that into a dataframe with nicely named columns.
Step7:  Everyone seems to start their speeches with "mr chairman" - how many speeches are there total, and how many don't mention "chairman" and how many mention neither "mr" nor "chairman"?
Step8:  What is the index of the speech which is the most thankful, a.k.a. includes the word 'thank' the most times?
Step9:  If I'm searching for China and trade, what are the top 3 speeches to read according to the CountVectoriser?
Step10:  Now what if I'm using a TfidfVectorizer?
Step11:  What's the content of the speeches? Here's a way to get them
Step12:  Now search for something else! Another two terms that might show up. elections and chaos? Whatever you thnik might be interesting.
Step13:  Enough of this garbage, let's cluster
Using a simple counting vectorizer, cluster the documents into eight categories, telling me what the top terms are per category.
Using a term frequency vectorizer, cluster the documents into eight categories, telling me what the top terms are per category.
Using a term frequency inverse document frequency vectorizer, cluster the documents into eight categories, telling me what the top terms are per category.
Step14:  Which one do you think works the best?
Step15:  Harry Potter time
I have a scraped collection of Harry Potter fanfiction at https
<ASSISTANT_TASK:>
Python Code:
# If you'd like to download it through the command line...
!curl -O http://www.cs.cornell.edu/home/llee/data/convote/convote_v1.1.tar.gz
# And then extract it through the command line...
!tar -zxf convote_v1.1.tar.gz
Explanation: Homework 14 (or so): TF-IDF text analysis and clustering
Hooray, we kind of figured out how text analysis works! Some of it is still magic, but at least the TF and IDF parts make a little sense. Kind of. Somewhat.
No, just kidding, we're professionals now.
Investigating the Congressional Record
The Congressional Record is more or less what happened in Congress every single day. Speeches and all that. A good large source of text data, maybe?
Let's pretend it's totally secret but we just got it leaked to us in a data dump, and we need to check it out. It was leaked from this page here.
End of explanation
# glob finds files matching a certain filename pattern
import glob
# Give me all the text files
paths = glob.glob('convote_v1.1/data_stage_one/development_set/*')
paths[:5]
len(paths)
Explanation: You can explore the files if you'd like, but we're going to get the ones from convote_v1.1/data_stage_one/development_set/. It's a bunch of text files.
End of explanation
speeches = []
for path in paths:
    with open(path) as speech_file:
        speech = {
            'pathname': path,
            'filename': path.split('/')[-1],
            'content': speech_file.read()
        }
    speeches.append(speech)
speeches_df = pd.DataFrame(speeches)
speeches_df.head()
Explanation: So great, we have 702 of them. Now let's import them.
End of explanation
speeches_df['content'].head(5)
Explanation: In class we had the texts variable. For the homework can just do speeches_df['content'] to get the same sort of list of stuff.
Take a look at the contents of the first 5 speeches
End of explanation
count_vectorizer = CountVectorizer(stop_words='english')
X=count_vectorizer.fit_transform(speeches_df['content'])
len(count_vectorizer.get_feature_names())
tokens_df=pd.DataFrame(X.toarray(), columns=count_vectorizer.get_feature_names())
tokens_df
Explanation: Doing our analysis
Use the sklearn package and a plain boring CountVectorizer to get a list of all of the tokens used in the speeches. If it won't list them all, that's ok! Make a dataframe with those terms as columns.
Be sure to include English-language stopwords
End of explanation
count_vectorizer = CountVectorizer(stop_words='english', max_features=100)
X=count_vectorizer.fit_transform(speeches_df['content'])
len(count_vectorizer.get_feature_names())
Explanation: Okay, it's far too big to even look at. Let's try to get a list of features from a new CountVectorizer that only takes the top 100 words.
End of explanation
tokens_df=pd.DataFrame(X.toarray(), columns=count_vectorizer.get_feature_names())
tokens_df
Explanation: Now let's push all of that into a dataframe with nicely named columns.
End of explanation
# 702 rows means 702 speeches, since each speech is a single string
len(tokens_df)
# if the speech doesnt contain a chairman, the column entry will be 0. so, 250 no-chairmain speeches. granted,
# we have no idea if they stared the speech with chairman or just mentioned him somewhere
len(tokens_df[tokens_df['chairman']==0])
# 76 times no mr or chairman. which means they must call the chairman just 'chairman' a lot. rude!
len(tokens_df[(tokens_df['mr']==0) & (tokens_df['chairman']==0)])
Explanation: Everyone seems to start their speeches with "mr chairman" - how many speeches are there total, and how many don't mention "chairman" and how many mention neither "mr" nor "chairman"?
End of explanation
# so speech index 375
tokens_df[tokens_df['thank']==tokens_df['thank'].max()].index
# lets look at the speech
speeches_df['content'][375]
# wow that was long. but its the most thankful one, so whatever.
Explanation: What is the index of the speech which is the most thankful, a.k.a. includes the word 'thank' the most times?
End of explanation
# i sorted by china here, on a lark. lets see if it holds true if i sort by trade
tokens_df[(tokens_df['china']>0) & (tokens_df['trade']>0)].sort_values(by='china', ascending=False)[['china', 'trade']].head(3)
tokens_df[(tokens_df['china']>0) & (tokens_df['trade']>0)].sort_values(by='trade', ascending=False)[['china', 'trade']].head(3)
# kind of! at any rate, speech 294 seems to be the most china and trade related. lets look at it!
speeches_df['content'][294]
# thats another super long speech, but it does seem to mostly be about trade and china.
Explanation: If I'm searching for China and trade, what are the top 3 speeches to read according to the CountVectoriser?
End of explanation
l2_vectorizer = TfidfVectorizer(stop_words='english', use_idf=True)
X = l2_vectorizer.fit_transform(speeches_df['content'])
tfidf_tokens_df = pd.DataFrame(X.toarray(), columns=l2_vectorizer.get_feature_names())
china_trade_df=pd.DataFrame([tfidf_tokens_df['china'], tfidf_tokens_df['trade'], tfidf_tokens_df['china'] + tfidf_tokens_df['trade']], index=["china", "trade", "china + trade"]).T
china_trade_df[china_trade_df.any(axis=1)].sort_values(by='china + trade', ascending=False).head(3)
# wow, that comes up with a totally different list of speeches. lets look at speech 636
speeches_df['content'][636]
# that one is very short by comparison and this time, its really only about china and trade
Explanation: Now what if I'm using a TfidfVectorizer?
End of explanation
# index 0 is the first speech, which was the first one imported.
paths[0]
# Pass that into 'cat' using { } which lets you put variables in shell commands
# that way you can pass the path to cat
!cat {paths[0]}
# i guess i probably should have read ahead to this part. oh well, dumping the index of the speeches_df 
# was still mostly readable
Explanation: What's the content of the speeches? Here's a way to get them:
End of explanation
election_chaos_df=pd.DataFrame([tfidf_tokens_df['election'], tfidf_tokens_df['chaos'], tfidf_tokens_df['election'] + tfidf_tokens_df['chaos']], index=["election", "chaos", "election + chaos"]).T
election_chaos_df[election_chaos_df.any(axis=1)].sort_values(by='chaos', ascending=False).head(10)
# i did the sort that way because i guess they dont talk about chaos much. lets look at that speech
!cat {paths[257]}
# thats pretty weak. i tried to come up with something spicier but i cant tell what years these are from.
# how about this?
clinton_welfare_df=pd.DataFrame([tfidf_tokens_df['clinton'], tfidf_tokens_df['welfare'], tfidf_tokens_df['clinton'] + tfidf_tokens_df['welfare']], index=["clinton", "welfare", "clinton + welfare"]).T
clinton_welfare_df[clinton_welfare_df.any(axis=1)].sort_values(by='clinton + welfare', ascending=False).head(10)
!cat {paths[346]}
# still pretty dull. oh well.
Explanation: Now search for something else! Another two terms that might show up. elections and chaos? Whatever you thnik might be interesting.
End of explanation
from sklearn.cluster import KMeans
count_vectorizer = CountVectorizer(stop_words='english')
X=count_vectorizer.fit_transform(speeches_df['content'])
number_of_clusters = 8
km = KMeans(n_clusters=number_of_clusters)
km.fit(X)
print("Top terms per cluster:")
order_centroids = km.cluster_centers_.argsort()[:, ::-1]
terms = count_vectorizer.get_feature_names()
for i in range(number_of_clusters):
    top_ten_words = [terms[ind] for ind in order_centroids[i, :5]]
    print("Cluster {}: {}".format(i, ' '.join(top_ten_words)))
tf_vectorizer = TfidfVectorizer(stop_words='english', use_idf=False)
X = tf_vectorizer.fit_transform(speeches_df['content'])
number_of_clusters = 8
km = KMeans(n_clusters=number_of_clusters)
km.fit(X)
print("Top terms per cluster:")
order_centroids = km.cluster_centers_.argsort()[:, ::-1]
terms = tf_vectorizer.get_feature_names()
for i in range(number_of_clusters):
    top_ten_words = [terms[ind] for ind in order_centroids[i, :5]]
    print("Cluster {}: {}".format(i, ' '.join(top_ten_words)))
tfidf_vectorizer = TfidfVectorizer(stop_words='english', use_idf=True)
X = tfidf_vectorizer.fit_transform(speeches_df['content'])
number_of_clusters = 8
km = KMeans(n_clusters=number_of_clusters)
km.fit(X)
print("Top terms per cluster:")
order_centroids = km.cluster_centers_.argsort()[:, ::-1]
terms = tfidf_vectorizer.get_feature_names()
for i in range(number_of_clusters):
    top_ten_words = [terms[ind] for ind in order_centroids[i, :5]]
    print("Cluster {}: {}".format(i, ' '.join(top_ten_words)))
Explanation: Enough of this garbage, let's cluster
Using a simple counting vectorizer, cluster the documents into eight categories, telling me what the top terms are per category.
Using a term frequency vectorizer, cluster the documents into eight categories, telling me what the top terms are per category.
Using a term frequency inverse document frequency vectorizer, cluster the documents into eight categories, telling me what the top terms are per category.
End of explanation
# well, it seems to be between results including wild horses and those including frivolous lawsuits.
# i prefer wild horses but the tfidf is probably the best representation of the actual document
Explanation: Which one do you think works the best?
End of explanation
paths = glob.glob('hp/hp/*')
paths[:5]
hp_fics = []
for path in paths:
    with open(path) as hp_file:
        hp_fic = {
            'pathname': path,
            'filename': path.split('/')[-1],
            'content': hp_file.read()
        }
    hp_fics.append(hp_fic)
hp_fics_df = pd.DataFrame(hp_fics)
hp_fics_df.head()
tfidf_vectorizer = TfidfVectorizer(stop_words='english', use_idf=True)
X = tfidf_vectorizer.fit_transform(hp_fics_df['content'])
number_of_clusters = 2
km = KMeans(n_clusters=number_of_clusters)
km.fit(X)
print("Top terms per cluster:")
order_centroids = km.cluster_centers_.argsort()[:, ::-1]
terms = tfidf_vectorizer.get_feature_names()
for i in range(number_of_clusters):
    top_ten_words = [terms[ind] for ind in order_centroids[i, :5]]
    print("Cluster {}: {}".format(i, ' '.join(top_ten_words)))
# i would say people are either writing about harry and hermione or lily and james
# ive never read harry potter, but i think that means either the harry generation or their parents generation?
# seems legit
Explanation: Harry Potter time
I have a scraped collection of Harry Potter fanfiction at https://github.com/ledeprogram/courses/raw/master/algorithms/data/hp.zip.
I want you to read them in, vectorize them and cluster them. Use this process to find out the two types of Harry Potter fanfiction. What is your hypothesis?
End of explanation
<END_TASK>