<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 Working with text
Sample from http
Step1:  Load in the training set of data
Step2:  Note target names not in same order as in the categories array
Count of documents
Step3:  Show the first 8 lines of text from one of the documents formated with line breaks
Step4:  Path to file on your machine
Step5:  Show the the targets categories of first 10 documents. As a list and show there names.
Step6:  Lets look at a document in the training data.
Step7:  Extracting features from text files
So for machine learning to be used text must be turned into numerical feature vectors.
What is a feature vector?
Each word is assigned an integer identifier
Each document is assigned an integer identifier
The results are stored in scipy.sparse matrices.
Tokenizing text with scikit-learn
Using CountVectorizer we load the training data into a spare matrix.
What is a spare matrix?
Step8:  Using a CountVectorizer method we can get the integer identifier of a word.
Step9:  With this identifier we can get the count of the word in a given document.
Step10:  What are two problems with using a word count in a document?
From occurrences to frequencies
$\text{Term Frequencies tf} = \text{occurrences of each word} / \text{total number of words}$
tf-idf is "Term Frequencies times Inverse Document Frequency"
Calculating tfidf
Step11:  .fit(..) to fit estimator to the data
.transform(..) to transform the count matrix to tf-idf
It is possible to merge the fit and transform stages using .fit_transform(..)
Calculate tfidf
Step12:  Training a classifier
So we now have features. We can train a classifier to try to predict the category of a post. First we will try the
naïve Bayes classifier.
Step13:  Here tfidf_transformer is used to classify
Step14:  We can see it get some right but not all.
Building a pipeline
Here we can put all the stages together in a pipeline. The names 'vect', 'tfidf' and 'clf' are arbitrary.
Step15:  Evaluation
Step16:  Try a support vector machine instead
Step17:  We can see the support vector machine got a higher number than naïve Bayes. What does it mean? We move on to metrics.
Using metrics
Classification report & Confusion matix
Here we will use a simple example to show classification reports and confusion matrices.
y_true is the test data
y_pred is the prediction
Step18:  Here we can see that the predictions
Step19:  In the confusion_matrix the labels give the order of the rows.
ant was correctly categorised twice and was never miss categorised
bird was correctly categorised once and was categorised as cat once
cat was correctly categorised twice and was categorised as an ant once
Step20:  Back to '20 newsgroups dataset'
Step21:  We can see where the 91% score came from.
Step22:  Now lets see the confusion matrix.
Step23:  So we can see the naïve Bayes classifier got a lot more correct in some cases but also included a higher proportion in the last category.
Step24:  We can see that atheism is miss categorised as Christian and science and medicine as computer graphics a high proportion of the time using the support vector machine.
Parameter tuning
Transformation and classifiers can have various parameters. Rather than manually tweaking each parameter in the pipeline it is possible to use grid search instead.
Here we try a couple of options for each stage. The more options the longer the grid search will take.
Step25:  Running the search on all the data will take a little while 10-30 seconds on a new ish desktop with 8 cores. If you don't want to wait that long uncomment the line with 
Step26:  Well that is a significant improvement. Lets use these new parameters.
Step27:  Why has this only give a .93 instead of .97?
http
Step28:  Moving on from that lets see where the improvements where made.
<ASSISTANT_TASK:>
Python Code:
categories = ['alt.atheism', 'soc.religion.christian','comp.graphics', 'sci.med']
Explanation: Working with text
Sample from http://scikit-learn.org/stable/tutorial/text_analytics/working_with_text_data.html
The data set is '20 newsgroups dataset' a dataset used for testing machine learning accuracy described at:
20 newsgroups dataset website.
We will be using this data to show scikit learn.
To make the samples run more quickly we will be limiting the example data set to just 4 categories.
End of explanation
from sklearn.datasets import fetch_20newsgroups
twenty_train = fetch_20newsgroups(subset='train',categories=categories, shuffle=True, random_state=42)
twenty_train.target_names
Explanation: Load in the training set of data
End of explanation
len(twenty_train.data)
Explanation: Note target names not in same order as in the categories array
Count of documents
End of explanation
print("\n".join(twenty_train.data[0].split("\n")[:8]))
Explanation: Show the first 8 lines of text from one of the documents formated with line breaks
End of explanation
twenty_train.filenames[0]
Explanation: Path to file on your machine
End of explanation
print(twenty_train.target[:10])
for t in twenty_train.target[:10]:
    print(twenty_train.target_names[t])
Explanation: Show the the targets categories of first 10 documents. As a list and show there names.
End of explanation
print("\n".join(twenty_train.data[0].split("\n")))
Explanation: Lets look at a document in the training data.
End of explanation
from sklearn.feature_extraction.text import CountVectorizer
count_vect = CountVectorizer()
X_train_counts = count_vect.fit_transform(twenty_train.data)
X_train_counts.shape
X_train_counts.__class__
Explanation: Extracting features from text files
So for machine learning to be used text must be turned into numerical feature vectors.
What is a feature vector?
Each word is assigned an integer identifier
Each document is assigned an integer identifier
The results are stored in scipy.sparse matrices.
Tokenizing text with scikit-learn
Using CountVectorizer we load the training data into a spare matrix.
What is a spare matrix?
End of explanation
count_vect.vocabulary_.get(u'application')
Explanation: Using a CountVectorizer method we can get the integer identifier of a word.
End of explanation
print("Word count for application in first document: {0} and last document: {1} ").format(
    X_train_counts[0, 5285], X_train_counts[2256, 5285])
count_vect.vocabulary_.get(u'subject')
print("Word count for email in first document: {0} and last document: {1} ").format(
    X_train_counts[0, 31077], X_train_counts[2256, 31077])
count_vect.vocabulary_.get(u'to')
print("Word count for email in first document: {0} and last document: {1} ").format(
    X_train_counts[0, 32493], X_train_counts[2256, 32493])
Explanation: With this identifier we can get the count of the word in a given document.
End of explanation
from sklearn.feature_extraction.text import TfidfTransformer
tf_transformer = TfidfTransformer(use_idf=False).fit(X_train_counts)
X_train_tfidf_2stage = tf_transformer.transform(X_train_counts)
X_train_tfidf_2stage.shape
Explanation: What are two problems with using a word count in a document?
From occurrences to frequencies
$\text{Term Frequencies tf} = \text{occurrences of each word} / \text{total number of words}$
tf-idf is "Term Frequencies times Inverse Document Frequency"
Calculating tfidf
End of explanation
tfidf_transformer = TfidfTransformer()
X_train_tfidf = tfidf_transformer.fit_transform(X_train_counts)
X_train_tfidf.shape
print("In first document tf-idf for application: {0} subject: {1} to: {2}").format(
    X_train_tfidf[0, 5285], X_train_tfidf[0, 31077], X_train_tfidf[0, 32493])
Explanation: .fit(..) to fit estimator to the data
.transform(..) to transform the count matrix to tf-idf
It is possible to merge the fit and transform stages using .fit_transform(..)
Calculate tfidf
End of explanation
from sklearn.naive_bayes import MultinomialNB
clf = MultinomialNB().fit(X_train_tfidf, twenty_train.target)
Explanation: Training a classifier
So we now have features. We can train a classifier to try to predict the category of a post. First we will try the
naïve Bayes classifier.
End of explanation
docs_new = ['God is love', 'Heart attacks are common', 'Disbelief in a proposition', 'Disbelief in a proposition means that one does not believe it to be true', 'OpenGL on the GPU is fast']
X_new_counts = count_vect.transform(docs_new)
X_new_tfidf = tfidf_transformer.transform(X_new_counts)
predicted = clf.predict(X_new_tfidf)
for doc, category in zip(docs_new, predicted):
     print('%r => %s' % (doc, twenty_train.target_names[category]))
Explanation: Here tfidf_transformer is used to classify
End of explanation
from sklearn.pipeline import Pipeline
text_clf_bayes = Pipeline([('vect', CountVectorizer()),
                     ('tfidf', TfidfTransformer()),
                     ('clf', MultinomialNB()),
])
text_clf_bayes_fit = text_clf_bayes.fit(twenty_train.data, twenty_train.target)
Explanation: We can see it get some right but not all.
Building a pipeline
Here we can put all the stages together in a pipeline. The names 'vect', 'tfidf' and 'clf' are arbitrary.
End of explanation
import numpy as np
twenty_test = fetch_20newsgroups(subset='test',
    categories=categories, shuffle=True, random_state=42)
docs_test = twenty_test.data
predicted_bayes = text_clf_bayes_fit.predict(docs_test)
np.mean(predicted_bayes == twenty_test.target)         
Explanation: Evaluation
End of explanation
from sklearn.linear_model import SGDClassifier
text_clf_svm = Pipeline([('vect', CountVectorizer()),
                         ('tfidf', TfidfTransformer()),
                         ('clf', SGDClassifier(loss='hinge', penalty='l2',
                                           alpha=1e-3, n_iter=5, random_state=42)),])
text_clf_svm_fit = text_clf_svm.fit(twenty_train.data, twenty_train.target)
predicted_svm = text_clf_svm_fit.predict(docs_test)
np.mean(predicted_svm == twenty_test.target)  
Explanation: Try a support vector machine instead
End of explanation
from sklearn import metrics
y_true = ["cat", "ant", "cat", "cat", "ant", "bird", "bird"]
y_pred = ["ant", "ant", "cat", "cat", "ant", "cat", "bird"]
print(metrics.classification_report(y_true, y_pred,
    target_names=["ant", "bird", "cat"]))
Explanation: We can see the support vector machine got a higher number than naïve Bayes. What does it mean? We move on to metrics.
Using metrics
Classification report & Confusion matix
Here we will use a simple example to show classification reports and confusion matrices.
y_true is the test data
y_pred is the prediction
End of explanation
metrics.confusion_matrix(y_true, y_pred, labels=["ant", "bird", "cat"])
Explanation: Here we can see that the predictions:
- found ant 3 times and should have found it twice hence precision of 0.67.
- never predicted ant when shouldn't have hence recall of 1.
- f1 source is the mean of precision and recall
- support of 2 meaning there were 2 in the true data set.
http://scikit-learn.org/stable/modules/generated/sklearn.metrics.precision_recall_fscore_support.html
Confusion matix
End of explanation
metrics.accuracy_score(y_true, y_pred, normalize=True, sample_weight=None)
Explanation: In the confusion_matrix the labels give the order of the rows.
ant was correctly categorised twice and was never miss categorised
bird was correctly categorised once and was categorised as cat once
cat was correctly categorised twice and was categorised as an ant once
End of explanation
print(metrics.classification_report(twenty_test.target, predicted_svm,
    target_names=twenty_test.target_names))
Explanation: Back to '20 newsgroups dataset'
End of explanation
# We got the evaluation score this way before:
print(np.mean(predicted_svm == twenty_test.target))
# We get the same results using metrics.accuracy_score
print(metrics.accuracy_score(twenty_test.target, predicted_svm, normalize=True, sample_weight=None))
Explanation: We can see where the 91% score came from.
End of explanation
print(twenty_train.target_names)
metrics.confusion_matrix(twenty_test.target, predicted_bayes)
Explanation: Now lets see the confusion matrix.
End of explanation
metrics.confusion_matrix(twenty_test.target, predicted_svm)
Explanation: So we can see the naïve Bayes classifier got a lot more correct in some cases but also included a higher proportion in the last category.
End of explanation
from sklearn.grid_search import GridSearchCV
parameters = {'vect__ngram_range': [(1, 1), (1, 2)],
              'tfidf__use_idf': (True, False),
              'clf__alpha': (1e-3, 1e-4),
}
gs_clf = GridSearchCV(text_clf_svm_fit, parameters, n_jobs=-1)
Explanation: We can see that atheism is miss categorised as Christian and science and medicine as computer graphics a high proportion of the time using the support vector machine.
Parameter tuning
Transformation and classifiers can have various parameters. Rather than manually tweaking each parameter in the pipeline it is possible to use grid search instead.
Here we try a couple of options for each stage. The more options the longer the grid search will take.
End of explanation
#gs_clf_fit = gs_clf.fit(twenty_train.data[:400], twenty_train.target[:400])
gs_clf_fit = gs_clf.fit(twenty_train.data, twenty_train.target)
best_parameters, score, _ = max(gs_clf_fit.grid_scores_, key=lambda x: x[1])
for param_name in sorted(parameters.keys()):
    print("%s: %r" % (param_name, best_parameters[param_name]))
score
Explanation: Running the search on all the data will take a little while 10-30 seconds on a new ish desktop with 8 cores. If you don't want to wait that long uncomment the line with :400 and comment out the other.
End of explanation
text_clf_svm_tuned = Pipeline([('vect', CountVectorizer(ngram_range=(1, 2))),
                     ('tfidf', TfidfTransformer(use_idf=True)),
                     ('clf', SGDClassifier(loss='hinge', penalty='l2',
                                           alpha=0.0001, n_iter=5, random_state=42)),
])
text_clf_svm_tuned_fit = text_clf_svm_tuned.fit(twenty_train.data, twenty_train.target)
predicted_tuned = text_clf_svm_tuned_fit.predict(docs_test)
metrics.accuracy_score(twenty_test.target, predicted_tuned, normalize=True, sample_weight=None)
Explanation: Well that is a significant improvement. Lets use these new parameters.
End of explanation
for x in gs_clf_fit.grid_scores_:
    print x[0], x[1], x[2]
Explanation: Why has this only give a .93 instead of .97?
http://scikit-learn.org/stable/modules/generated/sklearn.grid_search.GridSearchCV.html
End of explanation
print(metrics.classification_report(twenty_test.target, predicted_svm,
    target_names=twenty_test.target_names))
metrics.confusion_matrix(twenty_test.target, predicted_svm)
print(metrics.classification_report(twenty_test.target, predicted_tuned,
    target_names=twenty_test.target_names))
metrics.confusion_matrix(twenty_test.target, predicted_tuned)
Explanation: Moving on from that lets see where the improvements where made.
End of explanation
<END_TASK>