<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 Homework 14 (or so)
Step1:  You can explore the files if you'd like, but we're going to get the ones from convote_v1.1/data_stage_one/development_set/. It's a bunch of text files.
Step2:  So great, we have 702 of them. Now let's import them.
Step3:  In class we had the texts variable. For the homework can just do speeches_df['content'] to get the same sort of list of stuff.
Take a look at the contents of the first 5 speeches
Step4:  Doing our analysis
Use the sklearn package and a plain boring CountVectorizer to get a list of all of the tokens used in the speeches. If it won't list them all, that's ok! Make a dataframe with those terms as columns.
Be sure to include English-language stopwords
Step5:  Okay, it's far too big to even look at. Let's try to get a list of features from a new CountVectorizer that only takes the top 100 words.
Step6:  Now let's push all of that into a dataframe with nicely named columns.
Step7:  Everyone seems to start their speeches with "mr chairman" - how many speeches are there total, and many don't mention "chairman" and how many mention neither "mr" nor "chairman"?
Step8:  What is the index of the speech thank is the most thankful, a.k.a. includes the word 'thank' the most times?
Step9:  If I'm searching for China and trade, what are the top 3 speeches to read according to the CountVectoriser?
Step10:  Now what if I'm using a TfidfVectorizer?
Step11:  What's the content of the speeches? Here's a way to get them
Step12:  Now search for something else! Another two terms that might show up. elections and chaos? Whatever you thnik might be interesting.
Step13:  Enough of this garbage, let's cluster
Using a simple counting vectorizer, cluster the documents into eight categories, telling me what the top terms are per category.
Using a term frequency vectorizer, cluster the documents into eight categories, telling me what the top terms are per category.
Using a term frequency inverse document frequency vectorizer, cluster the documents into eight categories, telling me what the top terms are per category.
Step14:  Which one do you think works the best?
Harry Potter time
I have a scraped collection of Harry Potter fanfiction at https
Step15:  Vectorize
Count Vectorization
<ASSISTANT_TASK:>
Python Code:
# If you'd like to download it through the command line...
#!curl -O http://www.cs.cornell.edu/home/llee/data/convote/convote_v1.1.tar.gz
# And then extract it through the command line...
#!tar -zxf convote_v1.1.tar.gz
Explanation: Homework 14 (or so): TF-IDF text analysis and clustering
Hooray, we kind of figured out how text analysis works! Some of it is still magic, but at least the TF and IDF parts make a little sense. Kind of. Somewhat.
No, just kidding, we're professionals now.
Investigating the Congressional Record
The Congressional Record is more or less what happened in Congress every single day. Speeches and all that. A good large source of text data, maybe?
Let's pretend it's totally secret but we just got it leaked to us in a data dump, and we need to check it out. It was leaked from this page here.
End of explanation
# glob finds files matching a certain filename pattern
import glob
# Give me all the text files
paths = glob.glob('convote_v1.1/data_stage_one/development_set/*')
paths[:5]
len(paths)
Explanation: You can explore the files if you'd like, but we're going to get the ones from convote_v1.1/data_stage_one/development_set/. It's a bunch of text files.
End of explanation
speeches = []
for path in paths:
    with open(path) as speech_file:
        speech = {
            'pathname': path,
            'filename': path.split('/')[-1],
            'content': speech_file.read()
        }
    speeches.append(speech)
speeches_df = pd.DataFrame(speeches)
#speeches_df.head()
speeches_df['pathname'][0]
Explanation: So great, we have 702 of them. Now let's import them.
End of explanation
texts =speeches_df['content']
texts[:5]
Explanation: In class we had the texts variable. For the homework can just do speeches_df['content'] to get the same sort of list of stuff.
Take a look at the contents of the first 5 speeches
End of explanation
from sklearn.feature_extraction.text import CountVectorizer
count_vectorizer = CountVectorizer(stop_words='english')
Xc = count_vectorizer.fit_transform(texts)
Xc
Xc.toarray()
pd.DataFrame(Xc.toarray()).head(3)
Xc_feature= pd.DataFrame(Xc.toarray(), columns=count_vectorizer.get_feature_names())
Xc_feature.head(3)
Explanation: Doing our analysis
Use the sklearn package and a plain boring CountVectorizer to get a list of all of the tokens used in the speeches. If it won't list them all, that's ok! Make a dataframe with those terms as columns.
Be sure to include English-language stopwords
End of explanation
from nltk.stem.porter import PorterStemmer
porter_stemmer = PorterStemmer()
porter_stemmer = PorterStemmer()
def stemming_tokenizer(str_input):
    
    words = re.sub(r"[^A-Za-z]", " ", str_input).lower().split()
    words = [porter_stemmer.stem(word) for word in words]
    #print(words)
    return words
    
count_vectorizer = CountVectorizer(stop_words='english', tokenizer=stemming_tokenizer, max_features=100)
Xc100 = count_vectorizer.fit_transform(texts)
print(count_vectorizer.get_feature_names())
#count_vectorizer.get_feature_names()
Explanation: Okay, it's far too big to even look at. Let's try to get a list of features from a new CountVectorizer that only takes the top 100 words.
End of explanation
df_Xc = pd.DataFrame(Xc100.toarray(), columns=count_vectorizer.get_feature_names())
df_Xc.head(3)
Explanation: Now let's push all of that into a dataframe with nicely named columns.
End of explanation
df_Xc['act'].count()
df_Xc[df_Xc["chairman"]==0]['chairman'].count()
df_Xc[df_Xc["mr"]==0]['mr'].count()
total = df_Xc[df_Xc["mr"]==0]['mr'].count() + df_Xc[df_Xc["chairman"]==0]['chairman'].count()
print(total,"speaches in total do not mention neither 'mr' nor 'chairman'")
Explanation: Everyone seems to start their speeches with "mr chairman" - how many speeches are there total, and many don't mention "chairman" and how many mention neither "mr" nor "chairman"?
End of explanation
thank = df_Xc[df_Xc["thank"]!=0]
thank.head(3)
thank_column = thank['thank']
thank_column.sort(inplace=False, ascending=False).head(1)
Explanation: What is the index of the speech thank is the most thankful, a.k.a. includes the word 'thank' the most times?
End of explanation
china_trade = df_Xc['china'] + df_Xc['trade']
china_trade.sort(inplace=False, ascending=False).head(3)
Explanation: If I'm searching for China and trade, what are the top 3 speeches to read according to the CountVectoriser?
End of explanation
tfidf_vectorizer = TfidfVectorizer(stop_words='english', tokenizer=stemming_tokenizer, use_idf=False, norm='l1', max_features=100)
Xt = tfidf_vectorizer.fit_transform(texts)
pd.DataFrame(Xt.toarray(), columns=tfidf_vectorizer.get_feature_names()).head(3)
print(tfidf_vectorizer.get_feature_names())
# checking inverse term_vectorizer.get_feature_names()
l2_vectorizer = TfidfVectorizer(stop_words='english', tokenizer=stemming_tokenizer, use_idf=True, max_features=100)
Xl2 = l2_vectorizer.fit_transform(texts)
l2_df = pd.DataFrame(Xl2.toarray(), columns=l2_vectorizer.get_feature_names())
print(l2_vectorizer.get_feature_names())
Explanation: Now what if I'm using a TfidfVectorizer?
End of explanation
# index 0 is the first speech, which was the first one imported.
paths[0]
# Pass that into 'cat' using { } which lets you put variables in shell commands
# that way you can pass the path to cat
!echo {paths[0]}
!type a.text
Explanation: What's the content of the speeches? Here's a way to get them:
End of explanation
df_Xc.columns
congress_lawsuit = df_Xc['lawsuit'] + df_Xc['congress']
congress_lawsuit.sort(inplace=False, ascending=False).head(5)
pd.DataFrame([df_Xc['lawsuit'], df_Xc['congress'], df_Xc['lawsuit'] + df_Xc['congress']], index=["congress", "lawsuit", "congress + lawsuit"]).T
Explanation: Now search for something else! Another two terms that might show up. elections and chaos? Whatever you thnik might be interesting.
End of explanation
from sklearn.cluster import KMeans
#count vectorization Xc100 is a set of normalized for 100 top words
number_of_clusters = 8
km = KMeans(n_clusters=number_of_clusters)
km.fit(Xc100)
#count vectorization
print("Top terms per cluster:")
order_centroids = km.cluster_centers_.argsort()[:, ::-1]
terms = count_vectorizer.get_feature_names()
for i in range(number_of_clusters):
    top_ten_words = [terms[ind] for ind in order_centroids[i, :5]]
    print("Cluster {}: {}".format(i, ' '.join(top_ten_words)))
#texts
results = pd.DataFrame()
results['text'] = texts
results['category'] = km.labels_
results
#term vectorization Xc100 is a set of l1 term vect. for 100 top words
number_of_clusters = 8
kmt = KMeans(n_clusters=number_of_clusters)
kmt.fit(Xt)
print("Top terms per cluster:")
order_centroids = kmt.cluster_centers_.argsort()[:, ::-1]
terms_t = tfidf_vectorizer.get_feature_names()
for i in range(number_of_clusters):
    top_ten_words = [terms_t[ind] for ind in order_centroids[i, :5]]
    print("Cluster {}: {}".format(i, ' '.join(top_ten_words)))
#inverse term vectorization
number_of_clusters = 8
kml2 = KMeans(n_clusters=number_of_clusters)
kml2.fit(Xl2)
print("Top terms per cluster:")
order_centroids = km.cluster_centers_.argsort()[:, ::-1]
terms_r = l2_vectorizer.get_feature_names()
for i in range(number_of_clusters):
    top_ten_words = [terms_r[ind] for ind in order_centroids[i, :5]]
    print("Cluster {}: {}".format(i, ' '.join(top_ten_words)))
Explanation: Enough of this garbage, let's cluster
Using a simple counting vectorizer, cluster the documents into eight categories, telling me what the top terms are per category.
Using a term frequency vectorizer, cluster the documents into eight categories, telling me what the top terms are per category.
Using a term frequency inverse document frequency vectorizer, cluster the documents into eight categories, telling me what the top terms are per category.
End of explanation
import glob
paths = glob.glob('hp/hp/*')
paths[:5]
len(paths)
reviews = []
for path in paths:
    with open(path) as review_file:
        review = {
            'pathname': path,
            'filename': path.split('/')[-1],
            'content': review_file.read()
        }
    reviews.append(review)
reviews_df = pd.DataFrame(reviews)
reviews_df.head()
texts =reviews_df['content']
texts
Explanation: Which one do you think works the best?
Harry Potter time
I have a scraped collection of Harry Potter fanfiction at https://github.com/ledeprogram/courses/raw/master/algorithms/data/hp.zip.
I want you to read them in, vectorize them and cluster them. Use this process to find out the two types of Harry Potter fanfiction. What is your hypothesis?
End of explanation
from sklearn.feature_extraction.text import CountVectorizer
from nltk.stem.porter import PorterStemmer
porter_stemmer = PorterStemmer()
def stemming_tokenizer(str_input):
    
    words = re.sub(r"[^A-Za-z]", " ", str_input).lower().split()
    words = [porter_stemmer.stem(word) for word in words]
    #print(words)
    return words
count_vectorizer = CountVectorizer(stop_words="english", tokenizer=stemming_tokenizer, max_features = 1000)
Zc1000 = count_vectorizer.fit_transform(texts)
#print(count_vectorizer.get_feature_names())
from sklearn.cluster import KMeans
#count vectorization Zc100 is a set of normalized for 100 top words
number_of_clusters = 8
kmzc = KMeans(n_clusters=number_of_clusters)
kmzc.fit(Zc1000)
#count vectorization
print("Top terms per cluster:")
order_centroids = kmzc.cluster_centers_.argsort()[:, ::-1]
terms_zc = count_vectorizer.get_feature_names()
for i in range(number_of_clusters):
    top_ten_words = [terms_zc[ind] for ind in order_centroids[i, :5]]
    print("Cluster {}: {}".format(i, ' '.join(top_ten_words)))
#pd.DataFrame(Z.toarray(), columns=count_vectorizer.get_feature_names()).head(3)
tfidf_vectorizer = TfidfVectorizer(stop_words='english', tokenizer=stemming_tokenizer, use_idf=False, norm='l2')
Zt = tfidf_vectorizer.fit_transform(texts)
print(tfidf_vectorizer.get_feature_names())
#pd.DataFrame(Z.toarray(), columns=tfidf_vectorizer.get_feature_names()).head(3)
number_of_clusters = 8
kmt = KMeans(n_clusters=number_of_clusters)
kmt.fit(Zt)
#term vectorization
print("Top terms per cluster:")
order_centroids = kmt.cluster_centers_.argsort()[:, ::-1]
terms_zt = tfidf_vectorizer.get_feature_names()
for i in range(number_of_clusters):
    top_ten_words = [terms_zt[ind] for ind in order_centroids[i, :5]]
    print("Cluster {}: {}".format(i, ' '.join(top_ten_words)))
#reverse
tfidf_vectorizer = TfidfVectorizer(stop_words='english', tokenizer=stemming_tokenizer, use_idf=False, norm='l2')
Ztr = tfidf_vectorizer.fit_transform(texts)
print(tfidf_vectorizer.get_feature_names())
number_of_clusters = 8
kmtr = KMeans(n_clusters=number_of_clusters)
kmtr.fit(Ztr)
#term vectorization
print("Top terms per cluster:")
order_centroids = kmtr.cluster_centers_.argsort()[:, ::-1]
terms_ztr = tfidf_vectorizer.get_feature_names()
for i in range(number_of_clusters):
    top_ten_words = [terms_ztr[ind] for ind in order_centroids[i, :5]]
    print("Cluster {}: {}".format(i, ' '.join(top_ten_words)))
### genre it should be wa hermion s t hi and lili wa t s jame
Explanation: Vectorize
Count Vectorization
End of explanation
<END_TASK>