<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 Clustering documents
As we've discussed, the same principles that can be applied to clustering crime in two dimensions can also be applied to clustering doucments in much higher dimensional spaces. We'll demonstrate this concept using a selection of Jeb Bush's e-mails from his time serving as Florida's governor.
But first, we'll need to talk about what makes two documents "similar," which can be defined in a number of ways.
Step1:  We'll use those sample docs to start. Intuitively, you should be able to see that documents 0 and 1 have some similar elements ("dog," "jumped over," etc.) but document 2 is pretty different from the rest. Let's quantify that using two different distance measures
Step2:  Euclidean distance
We'll start by looking at Euclidean distance.
Euclidean distance is what you probably most commonly think of when you think of distance. It's the shortest path between two points on a plane. If you take two points and draw a line between them, the length of that line is the Euclidean distance.
<img src="http
Step3:  According to our Euclidean distance measure, document 0 and document 1 are 2.8 units apart, documents 0 and 2 are 4.1 units apart and documents 2 and 3 are 3.6 units apart. So this definitely captures the distances we're looking for. But in practice, there's another similarity measure that's more often used for looking at documents, known as cosine similarity.
Cosine similarity
Unlike Euclidean distance, which looks at the absolute distance between points, cosine similarity accomplishes something similar by looking at the angle between them on a plane, like so
Step4:  In practice, either one of these metrics can work for document similarity tasks. For now it's mostly important to know that there's more than one definition of similarity. Usually I start with cosine distance and test other metrics to see which work best for the task at hand.
Clustering e-mails
After that little digression into distance metrics, we can now move on to clustering real documents -- in this case, subject lines from a selection of Jeb Bush's e-mails. Conveniently, we can use basically the same code as we used for the crime clustering example to accomplish this task.
Step5:  You'll see that some clusters here make sense and others don't, which is partly a result of the size of the dataset we're using and the fact that subject lines are often too short to provide much meaningful clustering information. But you should also be able to see some clusters of documents that appear to make sense.
To try and get better results, let's raise the number of clusters in our dataset and also look at another vectorizing scheme known as TF-IDF, which is meant to minimize the value of words that occur frequently throughout the dataset.
<ASSISTANT_TASK:>
Python Code:
sample_docs = [
    'The quick brown fox jumped over the lazy dog',
    'The dog jumped over squirrel',
    'Four score and seven years ago'
]
Explanation: Clustering documents
As we've discussed, the same principles that can be applied to clustering crime in two dimensions can also be applied to clustering doucments in much higher dimensional spaces. We'll demonstrate this concept using a selection of Jeb Bush's e-mails from his time serving as Florida's governor.
But first, we'll need to talk about what makes two documents "similar," which can be defined in a number of ways.
End of explanation
# First we'll vectorize our documents, as we did last week
vectorizer = CountVectorizer()
features = vectorizer.fit_transform(sample_docs).toarray()
print features
Explanation: We'll use those sample docs to start. Intuitively, you should be able to see that documents 0 and 1 have some similar elements ("dog," "jumped over," etc.) but document 2 is pretty different from the rest. Let's quantify that using two different distance measures: Euclidean and Cosine.
End of explanation
# We'll use a helpful scikit-learn function to calculate their pairwise distances, starting with Euclidean
euclidean_distances = pairwise_distances(features, metric='euclidean')
print euclidean_distances
Explanation: Euclidean distance
We'll start by looking at Euclidean distance.
Euclidean distance is what you probably most commonly think of when you think of distance. It's the shortest path between two points on a plane. If you take two points and draw a line between them, the length of that line is the Euclidean distance.
<img src="http://i.stack.imgur.com/w7kJP.png">
End of explanation
cosine_distances = pairwise_distances(features, metric='cosine')
print cosine_distances
Explanation: According to our Euclidean distance measure, document 0 and document 1 are 2.8 units apart, documents 0 and 2 are 4.1 units apart and documents 2 and 3 are 3.6 units apart. So this definitely captures the distances we're looking for. But in practice, there's another similarity measure that's more often used for looking at documents, known as cosine similarity.
Cosine similarity
Unlike Euclidean distance, which looks at the absolute distance between points, cosine similarity accomplishes something similar by looking at the angle between them on a plane, like so:
<img src="https://engineering.aweber.com/wp-content/uploads/2013/02/4AUbj.png">
It is calculated via scikit-learn in a manner similar to Euclidean distance:
End of explanation
emails = open('data/jeb_subjects.csv').read().split('\n')
print emails [:100]
# As usual, we'll vectorize our documents first
vectorizer = CountVectorizer()
features = vectorizer.fit_transform(emails).toarray()
# Now we'll use k-means to try splitting them up into 20 groups
number_of_clusters = 20
kmeans = KMeans(n_clusters=number_of_clusters)
kmeans.fit(features)
print clusters_to_csv(kmeans.labels_, emails)
Explanation: In practice, either one of these metrics can work for document similarity tasks. For now it's mostly important to know that there's more than one definition of similarity. Usually I start with cosine distance and test other metrics to see which work best for the task at hand.
Clustering e-mails
After that little digression into distance metrics, we can now move on to clustering real documents -- in this case, subject lines from a selection of Jeb Bush's e-mails. Conveniently, we can use basically the same code as we used for the crime clustering example to accomplish this task.
End of explanation
# TF-IDF is invoked in the same way as a count vectorizer
vectorizer = TfidfVectorizer()
features = vectorizer.fit_transform(emails).toarray()
# Let's try 50 clusters
number_of_clusters = 50
kmeans = KMeans(n_clusters=number_of_clusters)
kmeans.fit(features)
print clusters_to_csv(kmeans.labels_, emails)
Explanation: You'll see that some clusters here make sense and others don't, which is partly a result of the size of the dataset we're using and the fact that subject lines are often too short to provide much meaningful clustering information. But you should also be able to see some clusters of documents that appear to make sense.
To try and get better results, let's raise the number of clusters in our dataset and also look at another vectorizing scheme known as TF-IDF, which is meant to minimize the value of words that occur frequently throughout the dataset.
End of explanation
<END_TASK>