<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 Homework 14 (or so)
Step1:  You can explore the files if you'd like, but we're going to get the ones from convote_v1.1/data_stage_one/development_set/. It's a bunch of text files.
Step2:  So great, we have 702 of them. Now let's import them.
Step3:  In class we had the texts variable. For the homework can just do speeches_df['content'] to get the same sort of list of stuff.
Take a look at the contents of the first 5 speeches
Step4:  Doing our analysis
Use the sklearn package and a plain boring CountVectorizer to get a list of all of the tokens used in the speeches. If it won't list them all, that's ok! Make a dataframe with those terms as columns.
Be sure to include English-language stopwords
Step5:  Okay, it's far too big to even look at. Let's try to get a list of features from a new CountVectorizer that only takes the top 100 words.
Step6:  Now let's push all of that into a dataframe with nicely named columns.
Step7:  Everyone seems to start their speeches with "mr chairman" - how many speeches are there total, and how many don't mention "chairman" and how many mention neither "mr" nor "chairman"?
Step8:  What is the index of the speech thank is the most thankful, a.k.a. includes the word 'thank' the most times?
Step9:  If I'm searching for China and trade, what are the top 3 speeches to read according to the CountVectoriser?
Step10:  Now what if I'm using a TfidfVectorizer?
Step11:  What's the content of the speeches? Here's a way to get them
Step12:  Now search for something else! Another two terms that might show up. elections and chaos? Whatever you thnik might be interesting.
Step13:  Enough of this garbage, let's cluster
Using a simple counting vectorizer, cluster the documents into eight categories, telling me what the top terms are per category.
Using a term frequency vectorizer, cluster the documents into eight categories, telling me what the top terms are per category.
Using a term frequency inverse document frequency vectorizer, cluster the documents into eight categories, telling me what the top terms are per category.
Step14:  Which one do you think works the best?
The last two seem to make more sense than the first one, telling from its cluster three. The last two are more human-readable. However human-readability ends with that distinction -- I can't tell which from the last two would be better, based on the top terms per cluster.
Harry Potter time
I have a scraped collection of Harry Potter fanfiction at https
<ASSISTANT_TASK:>
Python Code:
# If you'd like to download it through the command line...
!curl -O http://www.cs.cornell.edu/home/llee/data/convote/convote_v1.1.tar.gz
# And then extract it through the command line...
!tar -zxf convote_v1.1.tar.gz
Explanation: Homework 14 (or so): TF-IDF text analysis and clustering
Hooray, we kind of figured out how text analysis works! Some of it is still magic, but at least the TF and IDF parts make a little sense. Kind of. Somewhat.
No, just kidding, we're professionals now.
Investigating the Congressional Record
The Congressional Record is more or less what happened in Congress every single day. Speeches and all that. A good large source of text data, maybe?
Let's pretend it's totally secret but we just got it leaked to us in a data dump, and we need to check it out. It was leaked from this page here.
End of explanation
# glob finds files matching a certain filename pattern
import glob
# Give me all the text files
paths = glob.glob('convote_v1.1/data_stage_one/development_set/*')
paths[:5]
len(paths)
Explanation: You can explore the files if you'd like, but we're going to get the ones from convote_v1.1/data_stage_one/development_set/. It's a bunch of text files.
End of explanation
speeches = []
for path in paths:
    with open(path) as speech_file:
        speech = {
            'pathname': path,
            'filename': path.split('/')[-1],
            'content': speech_file.read()
        }
    speeches.append(speech)
speeches_df = pd.DataFrame(speeches)
speeches_df.head()
Explanation: So great, we have 702 of them. Now let's import them.
End of explanation
for item in speeches_df['content'].head(5):
    print("++++++++++++++++++++NEW SPEECH+++++++++++++++++++++")
    print(item)
    print("     ")
Explanation: In class we had the texts variable. For the homework can just do speeches_df['content'] to get the same sort of list of stuff.
Take a look at the contents of the first 5 speeches
End of explanation
c_vectorizer = CountVectorizer(stop_words='english')
x = c_vectorizer.fit_transform(speeches_df['content'])
x
df = pd.DataFrame(x.toarray(), columns=c_vectorizer.get_feature_names())
df
Explanation: Doing our analysis
Use the sklearn package and a plain boring CountVectorizer to get a list of all of the tokens used in the speeches. If it won't list them all, that's ok! Make a dataframe with those terms as columns.
Be sure to include English-language stopwords
End of explanation
#http://scikit-learn.org/stable/modules/generated/sklearn.feature_extraction.text.CountVectorizer.html
c2_vectorizer = CountVectorizer(stop_words='english', max_features=100)
y = c2_vectorizer.fit_transform(speeches_df['content'])
y
Explanation: Okay, it's far too big to even look at. Let's try to get a list of features from a new CountVectorizer that only takes the top 100 words.
End of explanation
new_df = pd.DataFrame(y.toarray(), columns=c2_vectorizer.get_feature_names())
#new_df
Explanation: Now let's push all of that into a dataframe with nicely named columns.
End of explanation
#http://stackoverflow.com/questions/15943769/how-to-get-row-count-of-pandas-dataframe
total_speeches = len(new_df.index)
print("In total there are", total_speeches, "speeches.")
wo_chairman = new_df[new_df['chairman']==0]['chairman'].count()
print(wo_chairman, "speeches don't mention 'chairman'")
wo_mr_chairman = new_df[(new_df['chairman']==0) & (new_df['mr']==0)]['chairman'].count()
print(wo_mr_chairman, "speeches don't mention neither 'chairman' nor 'mr'")
Explanation: Everyone seems to start their speeches with "mr chairman" - how many speeches are there total, and how many don't mention "chairman" and how many mention neither "mr" nor "chairman"?
End of explanation
#http://stackoverflow.com/questions/18199288/getting-the-integer-index-of-a-pandas-dataframe-row-fulfilling-a-condition
print("The speech with the most 'thank's has the index", np.where(new_df['thank']==new_df['thank'].max()))
Explanation: What is the index of the speech thank is the most thankful, a.k.a. includes the word 'thank' the most times?
End of explanation
china_trade_speeches = (new_df['china'] + new_df['trade']).sort_values(ascending = False).head(3)
china_trade_speeches
Explanation: If I'm searching for China and trade, what are the top 3 speeches to read according to the CountVectoriser?
End of explanation
porter_stemmer = PorterStemmer()
def stem_tokenizer(str_input):
    words = re.sub(r"[^A-Za-z0-9\-]", " ", str_input).lower().split()
    words = [porter_stemmer.stem(word) for word in words]
    return words
tfidf_vectorizer = TfidfVectorizer(stop_words='english', tokenizer=stem_tokenizer, use_idf=False, norm='l1', max_features=100)
X = tfidf_vectorizer.fit_transform(speeches_df['content'])
t_df = pd.DataFrame(X.toarray(), columns=tfidf_vectorizer.get_feature_names())
china_trade_speeches_v2 = (t_df['china'] + t_df['trade']).sort_values(ascending = False).head(3)
china_trade_speeches_v2
Explanation: Now what if I'm using a TfidfVectorizer?
End of explanation
# index 0 is the first speech, which was the first one imported.
paths[0]
# Pass that into 'cat' using { } which lets you put variables in shell commands
# that way you can pass the path to cat
print("++++++++++NEW SPEECH+++++++++")
!cat {paths[345]}
print("++++++++++NEW SPEECH+++++++++")
!cat {paths[336]}
print("++++++++++NEW SPEECH+++++++++")
!cat {paths[402]}
Explanation: What's the content of the speeches? Here's a way to get them:
End of explanation
new_df.columns
election_speeches = (new_df['discrimination'] + new_df['rights']).sort_values(ascending = False).head(3)
election_speeches
Explanation: Now search for something else! Another two terms that might show up. elections and chaos? Whatever you thnik might be interesting.
End of explanation
def new_stem_tokenizer(str_input):
    words = re.sub(r"[^A-Za-z0-9\-]", " ", str_input).lower().split()
    #With PorterStemmer implemented as above, the text was pretty crippled and hard to judge which made more sense.
    #that's why I have commented that line out for now
    
    #words = [porter_stemmer.stem(word) for word in words]
    return words
vectorizer_types = [
    {'name': 'CVectorizer', 'definition': CountVectorizer(stop_words='english', tokenizer=new_stem_tokenizer,  max_features=100)},
    {'name': 'TFVectorizer', 'definition': TfidfVectorizer(stop_words='english', tokenizer=new_stem_tokenizer,  max_features=100, use_idf=False)},
    {'name': 'TFVIDFVectorizer', 'definition': TfidfVectorizer(stop_words='english', tokenizer=new_stem_tokenizer,  max_features=100, use_idf=True)}
]
for vectorizer in vectorizer_types:
    X = vectorizer['definition'].fit_transform(speeches_df['content'])
    number_of_clusters = 8
    km = KMeans(n_clusters=number_of_clusters)
    km.fit(X)
    print("++++++++ Top terms per cluster -- using a", vectorizer['name'])
    order_centroids = km.cluster_centers_.argsort()[:, ::-1]
    terms = vectorizer['definition'].get_feature_names()
    for i in range(number_of_clusters):
        top_ten_words = [terms[ind] for ind in order_centroids[i, :7]]
        print("Cluster {}: {}".format(i, ' '.join(top_ten_words)))
Explanation: Enough of this garbage, let's cluster
Using a simple counting vectorizer, cluster the documents into eight categories, telling me what the top terms are per category.
Using a term frequency vectorizer, cluster the documents into eight categories, telling me what the top terms are per category.
Using a term frequency inverse document frequency vectorizer, cluster the documents into eight categories, telling me what the top terms are per category.
End of explanation
!curl -O https://github.com/ledeprogram/courses/raw/master/algorithms/data/hp.zip
import zipfile
import glob
potter_paths = glob.glob('hp/*')
potter_paths[:5]
potter = []
for path in potter_paths:
    with open(path) as potter_file:
        potter_text = {
            'pathname': path,
            'filename': path.split('/')[-1],
            'content': potter_file.read()
        }
    potter.append(potter_text)
potter_df = pd.DataFrame(potter)
potter_df.head()
vectorizer = TfidfVectorizer(stop_words='english', tokenizer=new_stem_tokenizer, use_idf=True)
X = vectorizer.fit_transform(potter_df['content'])
number_of_clusters = 2
km = KMeans(n_clusters=number_of_clusters)
km.fit(X)
print("Top terms per cluster")
order_centroids = km.cluster_centers_.argsort()[:, ::-1]
terms = vectorizer.get_feature_names()
for i in range(number_of_clusters):
    top_ten_words = [terms[ind] for ind in order_centroids[i, :7]]
    print("Cluster {}: {}".format(i, ' '.join(top_ten_words)))
Explanation: Which one do you think works the best?
The last two seem to make more sense than the first one, telling from its cluster three. The last two are more human-readable. However human-readability ends with that distinction -- I can't tell which from the last two would be better, based on the top terms per cluster.
Harry Potter time
I have a scraped collection of Harry Potter fanfiction at https://github.com/ledeprogram/courses/raw/master/algorithms/data/hp.zip.
I want you to read them in, vectorize them and cluster them. Use this process to find out the two types of Harry Potter fanfiction. What is your hypothesis?
End of explanation
<END_TASK>