<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 Modeling
ML Tasks
Step1:  Input
Step2:  Vectorizer
Step3:  Decided for BOW vectors, containing lemmatized words. BOW results (in this case) in better cluster performance than with tf-idf vectors. Lemmatization worked slightly better than stemming. (-> KElbow plots in plots/ dir).
Models
Step4:  Decided for 3 clusters, because of highest avg Silhoutte score compared to other cluster sizes.
Step5:  Nonetheless, the assignment isn't perfect. Cluster #1 looks good, but the many negative vals in cluster #0 & #1 suggest that there exist a cluster with more similar docs than in the actual assigned cluster. As a cluster size of 2 also leads to an inhomogen cluster and has a lower avg Silhoutte score, we go with the size of 3. Nevertheless, in general those findings suggest that the Sherlock Holmes stories should be represented in a single collection only.
Training
Step6:  Evaluation
Cluster density
Silhoutte coefficient
Step7:  Compared to original collections by Sir Arthur Conan Doyle
Step8:  Average Silhoutte coefficient is at least slightly positive and much better than the score of the original assignment (which is even negative). Success.
Visual Inspection
We come from the original assignment by Sir Arthur Conan Doyle...
Step9:  ... to the novel collection assignment
Step10:  Confirms the findings from the Silhoutte plot above (in the Models section), cluster #1 looks very coherent, cluster #2 is seperated and the two documents of cluster #0 fly somewhere around.
Nonetheless, compared to the original collection, this looks far better. Success.
Document-Cluster Assignment
Finally, we want to assign the Sherlock Holmes stories to the novel collection created by clustering, right?
Create artificial titles for the collections created from clusters.
Step11:  Let's see how the the books are differently assigned to collections by Sir Arthur Conan Doyle (Original Collection), respectively by the clustering algo (Novel Collection).
Step12:  Collections are uneven assigned. Cluster #1 is the predominant one. Looks like cluster #0 subsume the (rational) unassignable stories.
T-SNE plot eventually looks like that
<ASSISTANT_TASK:>
Python Code:
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
%matplotlib inline
Explanation: Modeling
ML Tasks
End of explanation
from sklearn.datasets import load_files
corpus = load_files("../data/")
doc_count = len(corpus.data)
print("Doc count:", doc_count)
assert doc_count is 56, "Wrong number of documents loaded, should be 56 (56 stories)"
Explanation: Input
End of explanation
from helpers.tokenizer import TextWrangler
from sklearn.feature_extraction.text import CountVectorizer
bow = CountVectorizer(strip_accents="ascii", tokenizer=TextWrangler(kind="lemma"))
X_bow = bow.fit_transform(corpus.data)
Explanation: Vectorizer
End of explanation
from sklearn.cluster import KMeans
kmeans = KMeans(n_jobs=-1, random_state=23)
from yellowbrick.cluster import KElbowVisualizer
viz = KElbowVisualizer(kmeans, k=(2, 28), metric="silhouette")
viz.fit(X_bow)
#viz.poof(outpath="plots/KElbow_bow_lemma_silhoutte.png")
viz.poof()
from yellowbrick.cluster import SilhouetteVisualizer
def plot_silhoutte_plots(max_n):
    for i in range(2, max_n + 1):
        plt.clf()
        n_cluster = i
        viz = SilhouetteVisualizer(KMeans(n_clusters=n_cluster, random_state=23))
        viz.fit(X_bow)
        path = "plots/SilhouetteViz" + str(n_cluster)
        viz.poof(outpath=path)
#plot_silhoutte_plots(28)
Explanation: Decided for BOW vectors, containing lemmatized words. BOW results (in this case) in better cluster performance than with tf-idf vectors. Lemmatization worked slightly better than stemming. (-> KElbow plots in plots/ dir).
Models
End of explanation
from yellowbrick.cluster import SilhouetteVisualizer
n_clusters = 3
model = KMeans(n_clusters=n_clusters, n_jobs=-1, random_state=23)
viz = SilhouetteVisualizer(model)
viz.fit(X_bow)
viz.poof()
Explanation: Decided for 3 clusters, because of highest avg Silhoutte score compared to other cluster sizes.
End of explanation
from sklearn.pipeline import Pipeline
pipe = Pipeline([("bow", bow),
                 ("kmeans", model)])
pipe.fit(corpus.data)
pred = pipe.predict(corpus.data)
Explanation: Nonetheless, the assignment isn't perfect. Cluster #1 looks good, but the many negative vals in cluster #0 & #1 suggest that there exist a cluster with more similar docs than in the actual assigned cluster. As a cluster size of 2 also leads to an inhomogen cluster and has a lower avg Silhoutte score, we go with the size of 3. Nevertheless, in general those findings suggest that the Sherlock Holmes stories should be represented in a single collection only.
Training
End of explanation
from sklearn.metrics import silhouette_score
print("Avg Silhoutte score:", silhouette_score(X_bow, pred), "(novel collections)")
Explanation: Evaluation
Cluster density
Silhoutte coefficient: [-1,1], where 1 is most dense and negative vals correspond to ill seperation.
End of explanation
print("AVG Silhoutte score", silhouette_score(X_bow, corpus.target), "(original collections)")
Explanation: Compared to original collections by Sir Arthur Conan Doyle:
End of explanation
from yellowbrick.text import TSNEVisualizer
# Map target names of original collections to target vals
collections_map = {}
for i, collection_name in enumerate(corpus.target_names):
    collections_map[i] = collection_name
# Plot
tsne_original = TSNEVisualizer()
labels = [collections_map[c] for c in corpus.target]
tsne_original.fit(X_bow, labels)
tsne_original.poof()
Explanation: Average Silhoutte coefficient is at least slightly positive and much better than the score of the original assignment (which is even negative). Success.
Visual Inspection
We come from the original assignment by Sir Arthur Conan Doyle...
End of explanation
# Plot
tsne_novel = TSNEVisualizer()
labels = ["c{}".format(c) for c in pipe.named_steps.kmeans.labels_]
tsne_novel.fit(X_bow, labels)
tsne_novel.poof()
Explanation: ... to the novel collection assignment:
End of explanation
# Novel titles, can be more creative ;>
novel_collections_map = {0: "The Unassignable Adventures of Cluster 0", 
                         1: "The Adventures of Sherlock Holmes in Cluster 1",
                         2: "The Case-Book of Cluster 2"}
Explanation: Confirms the findings from the Silhoutte plot above (in the Models section), cluster #1 looks very coherent, cluster #2 is seperated and the two documents of cluster #0 fly somewhere around.
Nonetheless, compared to the original collection, this looks far better. Success.
Document-Cluster Assignment
Finally, we want to assign the Sherlock Holmes stories to the novel collection created by clustering, right?
Create artificial titles for the collections created from clusters.
End of explanation
orig_assignment = [collections_map[c] for c in corpus.target]
novel_assignment = [novel_collections_map[p] for p in pred]
titles = [" ".join(f_name.split("/")[-1].split(".")[0].split("_")) 
          for f_name in corpus.filenames]
# Final df, compares original with new assignment
df_documents = pd.DataFrame([orig_assignment, novel_assignment], 
                            columns=titles, index=["Original Collection", "Novel Collection"]).T
df_documents.to_csv("collections.csv")
df_documents
df_documents["Novel Collection"].value_counts()
Explanation: Let's see how the the books are differently assigned to collections by Sir Arthur Conan Doyle (Original Collection), respectively by the clustering algo (Novel Collection).
End of explanation
tsne_novel_named = TSNEVisualizer(colormap="Accent")
tsne_novel_named.fit(X_bow, novel_assignment)
tsne_novel_named.poof(outpath="plots/Novel_Sherlock_Holmes_Collections.png")
Explanation: Collections are uneven assigned. Cluster #1 is the predominant one. Looks like cluster #0 subsume the (rational) unassignable stories.
T-SNE plot eventually looks like that:
End of explanation
<END_TASK>