<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 Classification metrics and Naive Bayes
We build an analytics model using text as our data, specifically trying to understand the sentiment of tweets about the company Apple. This is  a special classification problem, often called Sentiment Analysis.
The challenge is to see if we can correctly classify tweets as being negative, positive, or neutral about Apple.
Get the data
Step1:  It contains 1181 tweets (as text) and one manually labeled sentiment.
Step2:  2 means very positive, 0 is neutral and -2 is very negative
Step3:  Process the data
First of all, we clean the tweets, by lowering all the letters, removing punctuations and stop words and finally by tokenising and stemming it.
Step4:  Now we remove the stopwords. First we define which are the common words (stopwords) to be removed
Step5:  To remove a word from the corpus if that word is contained in our stopwords set, we need first to tokenise the corpus (i.e., split it into words or tokens)
Step6:  Create a Document-Term matrix
In text mining, it is important to create the document-term matrix (DTM) of the corpus we are interested in. A DTM is basically a matrix, with documents designated by rows and words by columns, that the elements are the counts or the weights (usually by tf-idf).
Step7:  Now we use the voctoriser to transform the corpus into a sparse matrix where each tweet has 1 if the feature is present in it or 0 if not.
Step8:  We put it into a data frame to use it in the classifier
Step9:  We start by splitting the tweets into training and test sets, as usual
Step10:  Train and test the classifier
Now we train the Naive Bayes classifier
Step11:  Metrics
Step12:  The classifier was correct 64% of times (not only if a tweet was negative but also if it was strongly negative or moderately negative).
A very useful metric is the confusion matrix that displays the predictions and the actual values in a matrix
Step13:  It's more clear if we visualise it as a heat map
Step14:  The numbers in the diagonal are all the times when the predicted sentiment for a tweet was the same as the actual sentiment. 
Now we can define accuracy as the sum of all the values in the diagonal divided by the total of the values.
The best accuracy would be 1.0 when all values are on the diagonal (no errors!), whereas the worst is 0.0 (nothing correct)!
Step15:  Which is the same value as above.  
A simple baseline
Now, how good is this accuracy?
Let's compare this to a simple baseline model that always predicts neutral (the most common tweets in the test dataset).
Step16:  This tells us that in our test dataset we have 178 observation with neutral sentiment and 118 with positive or negative tweets. 
So the accuracy of a baseline model that always predict non-negative tweets would be
Step17:  So our Naive Bayesian model does better than the simple baseline.
Predict the sentiment of a new tweet
The classifier can be applied to new tweets, of course, to predict their sentiment
Step18:  Ok. We try with two new tweets and see what we get, one positive and one negative
Step19:  Binary Classification
Now, this is a more generic case, where we have 5 classes as target.
The case you see more often is the binary one, with only two classes, which has some special characteristics and metrics.
Let's convert our target into a binary one
Step20:  We need to re-apply the classifier
Step21:  As you can see, there is no more classes 2, 0 or -2 now
Step22:  Of course is better, we have less classes to predict, less errors to make.
Let's see how the confusion matrix looks like
Step23:  In a two-class problem, we are often looking to discriminate between observations with a specific outcome, from normal observations. Such as a disease state or no disease state or spam versus no-spam.
One being the positive event and the other the no-event, the negative event.
In our case, let's say the negative event is the negative tweet and the positive event is the NON-negative tweet.  
These are basic terms used in binary classification
Step24:  Accuracy can be re-formulated as the ratio between the true events (positive and negative) and the total events
Step25:  Accuracy is not a reliable metric for the real performance of a classifier, because it will yield misleading results if the data set is unbalanced (that is, when the numbers of observations in different classes vary greatly).
Then you may consider additional metrics like Precision, Recall, F score (combined metric)
Step26:  Sensitivity is a real number between 0 and 1. A sensitivity of 1 means that ALL the Negative cases have been correctly classified. 
Specificity
Step27:  ROC (Receiver Operating Characteristic) curve
Until now, we have seen classification problems where we predict the target class directly.
Sometimes it can be more insightful or flexible to predict the probabilities for each class instead. From one side you will get an idea of how confident is the classifier for each class, on the other side you can use them to calibrate the threshold for how to interpret the predicted probabilities.
For example, in a binary classifier the default is to use a threshold of 0.5, meaning that a probability less than 0.5 is a negative outcome and a probability equal or over 0.5 is a positive outcome.
But this threshold can be adjusted to tune the behavior of the model for the specific problem, e.g. to reduce more of one or another type of error, as we have seen above.
Think about a classifier that predict if an event is a nuclear attack or not. Clearly you want to have as less as possible false alarms!
A diagnostic tools helpin in choosing the right threshold is the ROC curve.
This is the plot of the ‘True Positive Rate’ (Sensitivity) on the y-axis against the ‘False Positive Rate’ (1 minus Specificity) on the x-axis, at different classification thresholds between 0 and 1.
It captures all the thresholds simultaneously and the area under the ROC curve measures how well a parameter can distinguish between two groups. 
Threshold =0 is at the axis origin (0,0) while the threshold = 1 is at the top right end of the curve.
- high threshold
Step28:  The ROC curve is a useful tool for a few reasons
Step29:  Similarly, you can calculate the Negative Predictive Power
Step30:  The F1 score is the harmonic mean of the Precision & Sensitivity, and is used to indicate a balance between them. It ranges from 0 to 1; F1 Score reaches its best value at 1 (perfect precision & sensitivity) and worst at 0.
Step31:  What do we use the ROC for?
<ASSISTANT_TASK:>
Python Code:
import pandas as pd  # Start by importing the tweets data
X = pd.read_csv('../datasets/tweets.csv') 
X.shape
X.columns
X.info()
X.head(5)
Explanation: Classification metrics and Naive Bayes
We build an analytics model using text as our data, specifically trying to understand the sentiment of tweets about the company Apple. This is  a special classification problem, often called Sentiment Analysis.
The challenge is to see if we can correctly classify tweets as being negative, positive, or neutral about Apple.
Get the data
End of explanation
min(X.Avg)
max(X.Avg)
Explanation: It contains 1181 tweets (as text) and one manually labeled sentiment.
End of explanation
X.Avg.hist();
Explanation: 2 means very positive, 0 is neutral and -2 is very negative
End of explanation
corpusTweets = X.Tweet.tolist() # get a list of all tweets, then is easier to apply preprocessign to each item
# Convert to lower-case
corpusLowered = [s.lower() for s in corpusTweets]
corpusLowered[0:5]
# Remove punctuation
import re
corpusNoPunct = [re.sub(r'([^\s\w_]|_)+', ' ', s.strip()) for s in corpusLowered]
corpusNoPunct[0:5]
Explanation: Process the data
First of all, we clean the tweets, by lowering all the letters, removing punctuations and stop words and finally by tokenising and stemming it.
End of explanation
import os
def readStopwords():
    '''
    returns stopwords as strings
    
    Assume that a file called "stopwords.txt"
    exists in the folder
    '''    
    filename = "stopwords.txt"
    path = os.path.join("", filename)
    file = open(path, 'r')
    return file.read().splitlines()  # splitlines is used to remove newlines
stopWords = set(readStopwords())
"the" in stopWords  # quick test
stopWords.add("apple")
stopWords.add("appl")
stopWords.add("iphone")
stopWords.add("ipad")
stopWords.add("ipod")
stopWords.add("itunes")
stopWords.add("ios")
stopWords.add("http")
print ("apple" in stopWords)
print ("google" in stopWords)
Explanation: Now we remove the stopwords. First we define which are the common words (stopwords) to be removed:
End of explanation
# tokenise
corpusTokens = [s.split() for s in corpusNoPunct]
corpusTokens[0:3]
# Stem document 
from nltk import PorterStemmer
porter = PorterStemmer()
corpus = []
for tweet in corpusTokens:
    cleanTokens = [token for token in tweet if token not in stopWords] # a list of tokens
    stemmedTokens = [porter.stem(token) for token in cleanTokens]
    cleanTweet = ' '.join(stemmedTokens)
    corpus.append(cleanTweet)
corpus[0:5]
Explanation: To remove a word from the corpus if that word is contained in our stopwords set, we need first to tokenise the corpus (i.e., split it into words or tokens):
End of explanation
from sklearn.feature_extraction.text import CountVectorizer
cv = CountVectorizer(lowercase=False, max_features=500)
cv.fit(corpus)
'apple' in cv.vocabulary_  # a quick test
cv.get_feature_names()[0:20] # in alphabetical order
Explanation: Create a Document-Term matrix
In text mining, it is important to create the document-term matrix (DTM) of the corpus we are interested in. A DTM is basically a matrix, with documents designated by rows and words by columns, that the elements are the counts or the weights (usually by tf-idf).
End of explanation
bagOfWords = cv.transform(corpus)
bagOfWords
sum_words = bagOfWords.toarray().sum(axis=0)
words_freq = [(word, sum_words[idx]) for word, idx in cv.vocabulary_.items()]
words_freq =sorted(words_freq, key = lambda x: x[1], reverse=True)
words_freq[:10]
Explanation: Now we use the voctoriser to transform the corpus into a sparse matrix where each tweet has 1 if the feature is present in it or 0 if not.
End of explanation
df = pd.DataFrame(bagOfWords.toarray())
df.shape
df.info()
df.head(1)
Explanation: We put it into a data frame to use it in the classifier
End of explanation
import numpy.random
numpy.random.seed(100)  # just for reproducibility
from sklearn.model_selection import train_test_split
X.Avg = [int(round(a)) for a in X.Avg] # cluster target into 5 classes
X_train, X_test, y_train, y_test = train_test_split(df, X.Avg, test_size=0.25)
X_test.shape
Explanation: We start by splitting the tweets into training and test sets, as usual
End of explanation
from sklearn.naive_bayes import MultinomialNB
classifier = MultinomialNB()
classifier.fit(X_train, y_train)
predictions = classifier.predict(X_test)
predictions[0:100]
Explanation: Train and test the classifier
Now we train the Naive Bayes classifier
End of explanation
from sklearn import metrics
# Model Accuracy, how often is the classifier correct?
print("Accuracy: {:.2}".format(metrics.accuracy_score(y_test, predictions)))
Explanation: Metrics: accuracy and confusion matrix
We can check the quality of the predictions by using the scikit-learn metric, specifically the accuracy (we will see in a moment what i means):
End of explanation
mat = metrics.confusion_matrix(y_test, predictions)
mat
Explanation: The classifier was correct 64% of times (not only if a tweet was negative but also if it was strongly negative or moderately negative).
A very useful metric is the confusion matrix that displays the predictions and the actual values in a matrix:
End of explanation
import matplotlib.pyplot as plt
labels = ['strongly neg.', 'negative', 'neutral', 'positive', 'strongly pos.']
fig = plt.figure()
ax = fig.add_subplot(111)
cm = ax.matshow(mat)
# plot the title, use y to leave some space before the labels
plt.title("Confusion matrix - Tweets arranged by sentiment", y=1.2)
ax.set_xticklabels([''] + labels)
ax.set_yticklabels([''] + labels)
plt.setp(ax.get_xticklabels(), rotation=-30, ha="right",
             rotation_mode="anchor")
plt.xlabel("Predicted")
plt.ylabel("Actual")
# Loop over data dimensions and create text annotations.
for i in range(len(mat)):
    for j in range(len(mat)):
        text = ax.text(j, i, mat[i, j],
                       ha="center", va="center", color="w")
    # Create colorbar
fig.colorbar(cm);
Explanation: It's more clear if we visualise it as a heat map:
End of explanation
correctPredictions = sum(mat[i][i] for i in range(len(mat)))
correctPredictions
print("Accuracy: {:.2}".format(correctPredictions / len(y_test)))
Explanation: The numbers in the diagonal are all the times when the predicted sentiment for a tweet was the same as the actual sentiment. 
Now we can define accuracy as the sum of all the values in the diagonal divided by the total of the values.
The best accuracy would be 1.0 when all values are on the diagonal (no errors!), whereas the worst is 0.0 (nothing correct)!
End of explanation
neutralTweets = sum(1 for sentiment in y_test if sentiment == 0)  # neutral tweets in Test dataset
neutralTweets
len(y_test) - neutralTweets
Explanation: Which is the same value as above.  
A simple baseline
Now, how good is this accuracy?
Let's compare this to a simple baseline model that always predicts neutral (the most common tweets in the test dataset).
End of explanation
print("Accuracy baseline: {:.2}".format(neutralTweets / len(y_test)))
Explanation: This tells us that in our test dataset we have 178 observation with neutral sentiment and 118 with positive or negative tweets. 
So the accuracy of a baseline model that always predict non-negative tweets would be:
End of explanation
# for simplicity, it re-uses the vectorizer and the classifier without passing them
# as arguments. Industrialising it would mean to create a pipeline with
# vectoriser > classifier > label string
def predictSentiment(t):
    bow = cv.transform([t])
    prediction = classifier.predict(bow)
    if prediction == 0:
        return "Neutral"
    elif prediction > 0:
        return "Positive"
    else:
        return "Negative"
predictSentiment("I don't know what to think about apple!")
Explanation: So our Naive Bayesian model does better than the simple baseline.
Predict the sentiment of a new tweet
The classifier can be applied to new tweets, of course, to predict their sentiment:
End of explanation
predictSentiment("I love apple, its products are always the best, really!")
predictSentiment("Apple lost its mojo, I will never buy again an iphone better an Android")
Explanation: Ok. We try with two new tweets and see what we get, one positive and one negative
End of explanation
X.loc[X.Avg < 0] = -1 # negative sentiment
X.loc[X.Avg >= 0] = 1 # NON-negative sentiment
Explanation: Binary Classification
Now, this is a more generic case, where we have 5 classes as target.
The case you see more often is the binary one, with only two classes, which has some special characteristics and metrics.
Let's convert our target into a binary one:  a tweet can be either negative or not negative (i.e., positive or neutral).
First of all, we need to transform our original dataset to reduce the sentiment classes to only two classes.
End of explanation
X_train, X_test, y_train, y_test = train_test_split(df, X.Avg, test_size=0.25)
classifier = MultinomialNB()  # 0.77
classifier.fit(X_train, y_train)
predictionsTwo = classifier.predict(X_test)
predictionsTwo[0:100]
Explanation: We need to re-apply the classifier
End of explanation
# Model Accuracy, how often is the classifier correct?
print("Accuracy: {:.2}".format(metrics.accuracy_score(y_test, predictionsTwo)))
Explanation: As you can see, there is no more classes 2, 0 or -2 now
End of explanation
matBinary = metrics.confusion_matrix(y_test, predictionsTwo)
matBinary
labels = ['negative', 'NOT negative']
fig = plt.figure()
ax = fig.add_subplot(111)
cm = ax.matshow(matBinary)
# plot the title, use y to leave some space before the labels
plt.title("Confusion matrix - Tweets arranged by sentiment", y=1.2)
ax.set_xticklabels([''] + labels)
ax.set_yticklabels([''] + labels)
plt.setp(ax.get_xticklabels(), rotation=-30, ha="right",
             rotation_mode="anchor")
plt.xlabel("Predicted")
plt.ylabel("Actual")
# Loop over data dimensions and create text annotations.
for i in range(len(matBinary)):
    for j in range(len(matBinary)):
        text = ax.text(j, i, matBinary[i, j],
                       ha="center", va="center", color="w")
    # Create colorbar
fig.colorbar(cm);
Explanation: Of course is better, we have less classes to predict, less errors to make.
Let's see how the confusion matrix looks like:
End of explanation
tn, fp, fn, tp = matBinary.ravel()
print("True Negatives: ",tn)
print("False Positives: ",fp)
print("False Negatives: ",fn)
print("True Positives: ",tp)
Explanation: In a two-class problem, we are often looking to discriminate between observations with a specific outcome, from normal observations. Such as a disease state or no disease state or spam versus no-spam.
One being the positive event and the other the no-event, the negative event.
In our case, let's say the negative event is the negative tweet and the positive event is the NON-negative tweet.  
These are basic terms used in binary classification:
“true positive” for correctly predicted event values (in our scenario the non-negative tweets: positive or neutral).
“true negative” for correctly predicted no-event values (in our scenario the negative tweets).
“false positive” for incorrectly predicted event values. In Hypothesis Testing it is also known as Type 1 error or the incorrect rejection of Null Hypothesis.
“false negative” for incorrectly predicted no-event values. It is also known as Type 2 error, which leads to the failure in rejection of Null Hypothesis.
End of explanation
Accuracy = (tn+tp)/(tp+tn+fp+fn) 
print("Accuracy: {:.2f}".format(Accuracy))
Explanation: Accuracy can be re-formulated as the ratio between the true events (positive and negative) and the total events:
End of explanation
Sensitivity = tp/(tp+fn) 
print("Sensitivity {:0.2f}".format(Sensitivity))
Explanation: Accuracy is not a reliable metric for the real performance of a classifier, because it will yield misleading results if the data set is unbalanced (that is, when the numbers of observations in different classes vary greatly).
Then you may consider additional metrics like Precision, Recall, F score (combined metric):
Sensitivity or Recall
It is the ‘Completeness’, ability of the model to identify all relevant instances, True Positive Rate, aka Sensitivity.
Imagine a scenario where your focus is to have the least False Negatives, ofr example if you are trying to predict if an email is a spam or not, you don’t want authentic messages to be wrongly classified as spam. Then Sensitivity can come to rescue:
End of explanation
#Specificity 
Specificity = tn/(tn+fp)
print("Specificity {:0.2f}".format(Specificity))
Explanation: Sensitivity is a real number between 0 and 1. A sensitivity of 1 means that ALL the Negative cases have been correctly classified. 
Specificity
End of explanation
probs = classifier.predict_proba(X_test) # get the probabilities
preds = probs[:,1]  ## keep probabilities for the positive outcome only
fpr, tpr, threshold = metrics.roc_curve(y_test, preds)  # calculate roc 
roc_auc = metrics.auc(fpr, tpr)  # calculate AUC
plt.title('Receiver Operating Characteristic')
plt.plot(fpr, tpr, 'b', label = 'AUC = %0.2f' % roc_auc)
plt.plot([0, 1], [0, 1],'r--')  # plot random guessing
plt.legend(loc = 'lower right')
plt.xlim([0, 1])
plt.ylim([0, 1])
plt.ylabel('True Positive Rate')
plt.xlabel('False Positive Rate')
plt.show()
Explanation: ROC (Receiver Operating Characteristic) curve
Until now, we have seen classification problems where we predict the target class directly.
Sometimes it can be more insightful or flexible to predict the probabilities for each class instead. From one side you will get an idea of how confident is the classifier for each class, on the other side you can use them to calibrate the threshold for how to interpret the predicted probabilities.
For example, in a binary classifier the default is to use a threshold of 0.5, meaning that a probability less than 0.5 is a negative outcome and a probability equal or over 0.5 is a positive outcome.
But this threshold can be adjusted to tune the behavior of the model for the specific problem, e.g. to reduce more of one or another type of error, as we have seen above.
Think about a classifier that predict if an event is a nuclear attack or not. Clearly you want to have as less as possible false alarms!
A diagnostic tools helpin in choosing the right threshold is the ROC curve.
This is the plot of the ‘True Positive Rate’ (Sensitivity) on the y-axis against the ‘False Positive Rate’ (1 minus Specificity) on the x-axis, at different classification thresholds between 0 and 1.
It captures all the thresholds simultaneously and the area under the ROC curve measures how well a parameter can distinguish between two groups. 
Threshold =0 is at the axis origin (0,0) while the threshold = 1 is at the top right end of the curve.
- high threshold: means high specificity and low sensitivity
- Low threshold: means low specificity and high sensitivity  
Put another way, it plots the false alarm rate versus the hit rate.  
Let's see an example using our binary classification above.
First, you need probabilities to create the ROC curve.
End of explanation
# Precision 
Precision = tp/(tp+fp) 
print("Precision or Positive Predictive Power: {:0.2f}".format(Precision))
Explanation: The ROC curve is a useful tool for a few reasons:
The curves of different models can be compared directly in general or for different thresholds.
The area under the curve (AUC) can be used as a summary of the model skill.
A random guessing classifier (the red line above) has an Area Under the Curve (often referred as AUC) of 0.5, while AUC for a perfect classifier is equal to 1.
In general AUC of above 0.8 is considered "good".
Looking at the ROC curve you can choose a threshold that gives a desirable balance between the:
- cost of failing to detect positive
- cost of raising false alarms
Precision and F1 score metrics
Called also Positive Predictive Power, the Precision measures somehow how "exact" it is, i.e.  the ability of the model to return only relevant instances. If your use case/problem statement involves minimising the False Positives then Precision is something you need:
End of explanation
# Negative Predictive Value
print("Negative predictive Power: {:0.2f}".format(tn / (tn+fn)))
Explanation: Similarly, you can calculate the Negative Predictive Power:
End of explanation
# F1 Score
f1 = (2 * Precision * Sensitivity) / (Precision + Sensitivity)
print("F1 Score {:0.2f}".format(f1))
Explanation: The F1 score is the harmonic mean of the Precision & Sensitivity, and is used to indicate a balance between them. It ranges from 0 to 1; F1 Score reaches its best value at 1 (perfect precision & sensitivity) and worst at 0.
End of explanation
classifierTuned = MultinomialNB(class_prior=[.4, 0.6]) # try to max specificity
classifierTuned.fit(X_train, y_train)
predictionsTuned = classifierTuned.predict(X_test)
matTuned = metrics.confusion_matrix(y_test, predictionsTuned)
matTuned
tn, fp, fn, tp = matTuned.ravel()
Accuracy = (tn+tp)/(tp+tn+fp+fn) 
print("Accuracy: {:.2f}".format(Accuracy)) # it was 0.79
Sensitivity = tp/(tp+fn) 
print("Sensitivity {:0.2f}".format(Sensitivity)) #it was 0.9
Specificity = tn/(tn+fp)
print("Specificity {:0.2f}".format(Specificity)) # it was 0.53
Explanation: What do we use the ROC for?
End of explanation
<END_TASK>