<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 Assignment 7 - Part A
Student UW NetID 
Step1:  Load Dataset
Step2:  Create Train and Test Data  [from categories-medical and automobiles]
Step3:  Explore the data
Step4:  Pre-process Data
Step5:  Transform Data (Vectorize)
Step6:  There are 18000+ features for each document. And on average, 87 out of 18000 features are non-zeros. This is a sparse matrix
Train Classifier
Step7:  Evaluate Classifier
Step8:  Interpretation
Step10:  Pretty Print Confusion Matrix
Step12:  Interpretation
Step13:  Interpretation
<ASSISTANT_TASK:>
Python Code:
import re
from time import time
import string
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
from pprint import pprint
#Sklearn Imports
from sklearn import metrics
from sklearn.datasets import fetch_20newsgroups
from sklearn import preprocessing
from sklearn.pipeline import Pipeline
from sklearn.feature_extraction.text import TfidfVectorizer, TfidfTransformer
from sklearn.naive_bayes import MultinomialNB
from sklearn.model_selection import GridSearchCV
from sklearn.metrics import confusion_matrix, precision_recall_curve, roc_auc_score, auc
from nltk import PorterStemmer
from nltk.corpus import stopwords
from nltk.tokenize import sent_tokenize, word_tokenize
import nltk
nltk.download('stopwords') #download the latest stopwords
Explanation: Assignment 7 - Part A
Student UW NetID : vaibhavi@uw.edu
Student Name     : Vaibhavi Rangarajan
Import Libraries
End of explanation
all_newsgroups= fetch_20newsgroups()
pprint(list(all_newsgroups.target_names))
Explanation: Load Dataset
End of explanation
cats = ['sci.med' , 'rec.motorcycles']
newsgroups_train = fetch_20newsgroups(subset='train', categories=cats, remove=('headers', 'footers', 'quotes'))
newsgroups_test = fetch_20newsgroups(subset='test', categories=cats, remove=('headers', 'footers', 'quotes'))
print("Categories to classify\n-----------------------")
print(list(newsgroups_train.target_names))
print("TRAIN DATA\n---------------")
print("Data Type:", type(newsgroups_train))
print("%d documents" % len(newsgroups_train.filenames))
print("%d categories" % len(newsgroups_train.target_names))
print("X shape :", newsgroups_train.filenames.shape)
print("Y shape :",newsgroups_train.target.shape)
print("Y head  :", newsgroups_train.target[:10])
print("TEST DATA\n---------------")
print("Data Type:", type(newsgroups_test))
print("%d documents" % len(newsgroups_test.filenames))
print("%d categories" % len(newsgroups_test.target_names))
print("X shape :", newsgroups_test.filenames.shape)
print("Y shape :",newsgroups_test.target.shape)
print("Y head  :", newsgroups_test.target[:10])
Explanation: Create Train and Test Data  [from categories-medical and automobiles]
End of explanation
print(newsgroups_train.data[0])
print(newsgroups_test.data[0])
print(type(newsgroups_test.data))
print(type(newsgroups_test.data[0]))
Explanation: Explore the data
End of explanation
train_labels = newsgroups_train.target #0, 1 array
#print(train_labels)
test_labels = newsgroups_test.target
#print(test_labels)
RE_PREPROCESS = r'\W+|\d+' #the regular expressions that matches all non-characters
#train_corpus = np.array( [re.sub(RE_PREPROCESS, ' ', text).lower() for text in df_train.jobDescription.values])
#test_corpus = np.array( [re.sub(RE_PREPROCESS, ' ', text).lower() for text in df_test.jobDescription.values])
labels = np.append(train_labels, test_labels)
Explanation: Pre-process Data
End of explanation
vectorizer = TfidfVectorizer()
vectors_train = vectorizer.fit_transform(newsgroups_train.data)
vectors_train.shape
vectors_train.nnz / float(vectors_train.shape[0])
vectors_test = vectorizer.transform(newsgroups_test.data)
Explanation: Transform Data (Vectorize)
End of explanation
clf = MultinomialNB(alpha=.01)
clf.fit(vectors_train, newsgroups_train.target)
Explanation: There are 18000+ features for each document. And on average, 87 out of 18000 features are non-zeros. This is a sparse matrix
Train Classifier
End of explanation
y_true = newsgroups_test.target
y_pred = clf.predict(vectors_test)
metrics.f1_score(y_true, y_pred, average='macro')
Explanation: Evaluate Classifier
End of explanation
cm = confusion_matrix(y_true, y_pred)
Explanation: Interpretation:  An F-1 score of 0.94 is high. Our model is performant.
Plot Confusion Matrix
End of explanation
def print_cm(cm, labels, hide_zeroes=False, hide_diagonal=False, hide_threshold=None):
    pretty print for confusion matrixes
    columnwidth = max([len(x) for x in labels] + [5])  # 5 is value length
    empty_cell = " " * columnwidth
    # Print header
    print("    " + empty_cell, end=" ")
    for label in labels:
        print("%{0}s".format(columnwidth) % label, end=" ")
    print()
    
    # Print rows
    for i, label1 in enumerate(labels):
        print("    %{0}s".format(columnwidth) % label1, end=" ")
        for j in range(len(labels)):
            cell = "%{0}.1f".format(columnwidth) % cm[i, j]
            if hide_zeroes:
                cell = cell if float(cm[i, j]) != 0 else empty_cell
            if hide_diagonal:
                cell = cell if i != j else empty_cell
            if hide_threshold:
                cell = cell if cm[i, j] > hide_threshold else empty_cell
            print(cell, end=" ")
        print()
print_cm(cm, labels = ['Automobiles', 'Medical'])
pd.crosstab(y_true, y_pred, rownames=['True'], colnames=['Predicted'], margins=True)
Explanation: Pretty Print Confusion Matrix
End of explanation
def plot_precision_recall(y_true,y_score):
    
    Plot a precision recall curve
    
    Parameters
    ----------
    y_true: ls
        ground truth labels
    y_score: ls
        score output from model
    
    precision_curve, recall_curve, pr_thresholds = precision_recall_curve(y_true,y_score[:,1])
    plt.plot(recall_curve, precision_curve)
    plt.xlabel('Recall')
    plt.ylabel('Precision')
    auc_val = auc(recall_curve,precision_curve)
    print('AUC-PR: {0:1f}'.format(auc_val))
    plt.show()
    plt.clf()
y_score = clf.predict_proba(vectors_test)
plot_precision_recall(y_true, y_score)
Explanation: Interpretation:
From 398 Automobile related articles, we classifier 385 Correctly.
From 396 Medicine related articles, we classified, 363 Correctly.
Plot: Precision-Recall Curve
End of explanation
#Params - NOT tuned
ANALYZER = "word" #unit of features are single words rather then phrases of words 
STRIP_ACCENTS = 'unicode'
TOKENIZER = None
MAX_DF = (1.0)  # Exclude words that have a frequency greater than the threshold 
STOP_WORDS = (stopwords.words('english'), None)
#Params - TUNED
NGRAM_RANGE = ((0,1), (0,2)) #Range for pharases of words
MIN_DF = (0, 0.01) # Exclude words that have a frequency less than the threshold
ALPHA = (0.01, 0.1, 1)
pipeline = Pipeline([('tfidf', TfidfVectorizer()), ('clf', MultinomialNB())])
# uncommenting more parameters will give better exploring power but will
# increase processing time in a combinatorial way
parameters = {
    'tfidf__ngram_range':NGRAM_RANGE,
    'tfidf__min_df':MIN_DF,
    'clf__alpha': ALPHA,
}
def optimize_pipeline(pipeline):
    # multiprocessing requires the fork to happen in a __main__ protected
    # block
    # find the best parameters for both the feature extraction and the
    # classifier
    grid_search = GridSearchCV(pipeline, parameters, n_jobs=-1, verbose=True)
    print("Performing grid search...")
    print("pipeline:", [name for name, _ in pipeline.steps])
    print("parameters:")
    pprint(parameters)
    t0 = time()
    grid_search.fit(newsgroups_train.data, newsgroups_train.target)
    print("done in %0.3fs" % (time() - t0))
    print()
    print("Best score: %0.3f" % grid_search.best_score_)
    print("Best parameters set:")
    best_parameters = grid_search.best_estimator_.get_params()
    for param_name in sorted(parameters.keys()):
        print("\t%s: %r" % (param_name, best_parameters[param_name]))
optimize_pipeline(pipeline)
Explanation: Interpretation:
The area under the curve is 0.98, just shy of the ideal 1.0.
The trained classifier is extending to the test set well.
Improve:  Grid Search CV for Classifier
Let's play with parameters for the TFIDF Vectorizer, and Alpha (Laplace smoothing) for the Naive Bayes Classifier
http://scikit-learn.org/stable/modules/generated/sklearn.feature_extraction.text.TfidfVectorizer.html
http://scikit-learn.org/stable/modules/generated/sklearn.naive_bayes.MultinomialNB.html
End of explanation
<END_TASK>