<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 YouTube Spam Collection Data Set (Part 2)
Source
Step1:  <a id='section3b'></a>
Read in the data from the first CSV alone
Step2:  <a id='section3c'></a>
Data cleanup
Step3:  <a id='section3d'></a>
Split the data
Step4:  <a id='section3e'></a>
Transform the data
Step5:  <a id='section3f'></a>
Build the model
In this step, we will build 6 models, and pick the one with the best accuracy score
Step6:  Model selection
Based on accuracy scores, the best algorithm is the Decision Tree Classifier. Logistic Regression and AdaBoost Classifier also performed very well. We will choose Decision Tree as our model, and look to tune it.
Step7:  It looks like we were able to eke out some improvement in the performance. The Decision Tree Classifier seems to perform best with the min_samples_split set to 7. We will use this for our final model. Note that the default values for 'criterion' and 'splitter' seem to be part of the best performing set of parameters.
<a id='section3g'></a>
Run the prediction
Step8:  <a id='section3h'></a>
Score the prediction
Step9:  Some of the false negatives seem like they should have been marked as spam, so it is interesting that the model missed these. We may need to tune our vectorizer and/or attempt some other classifiers.
Let us check the area under the ROC curve.
<ASSISTANT_TASK:>
Python Code:
# Import modules
import numpy as np
import pandas as pd
Explanation: YouTube Spam Collection Data Set (Part 2)
Source: UCI Machine Learning Repository
Original Source: YouTube Spam Collection v. 1
Alberto, T.C., Lochter J.V., Almeida, T.A. Filtragem Automática de Spam nos Comentários do YouTube. Anais do XII Encontro Nacional de Inteligência Artificial e Computacional (ENIAC'15), Natal, RN, Brazil, 2015. (preprint)
Alberto, T.C., Lochter J.V., Almeida, T.A. TubeSpam: Comment Spam Filtering on YouTube. Proceedings of the 14th IEEE International Conference on Machine Learning and Applications (ICMLA'15), 1-6, Miami, FL, USA, December, 2015. (preprint)
Contents
1 Data Set Description
2 Approach
3 Solution
3a Import modules
3b Read the data set
3c Data cleanup
3d Split the data
3e Transform the data
3f Build the model
3g Run predictions
3h Score the prediction
4 Summary
<a id='section1'></a>
1. Data Set Description
From the description accompanying the data set, "the samples were extracted from the comments section of five videos that were among the 10 most viewed on YouTube during the collection period."
The data is available in five distinct data sets, and the data is classified as 1 for "spam" and 0 for "ham"
<a id='section2'></a>
2. Approach
Since the data set is split across five data sets, we will take two passes at the data. This is the second pass.
In the (optional) first pass, we considered only the Psy data set, as a way to wrap our hands around the problem. The notebook for this can be accessed here.
Our second pass will involve merging all five data sets and then running the classification on the combined data set. In this round, we will also tune the model and the vectorizer to eke out some improvements.
<a id='section3'></a>
3. Solution
<a id='section3a'></a>
Import initial set of modules
End of explanation
# Read the data set; print the first few rows
files = ['data\\Youtube01-Psy.csv', 'data\\Youtube02-KatyPerry.csv', 'data\\Youtube03-LMFAO.csv', 
         'data\\Youtube04-Eminem.csv', 'data\\Youtube05-Shakira.csv']
df = pd.DataFrame()
for file in files:
    df = df.append(pd.read_csv(file))
    
df.head()
Explanation: <a id='section3b'></a>
Read in the data from the first CSV alone
End of explanation
# Check for missing values
df.info()
# Looks like there are missing values in the DATE column, but it is not a column of interest. Let's proceed.
# Of the five columns, the only relevant columns for spam/ham classification are the CONTENT and CLASS columns.
# We will use just these two columns. But first, let's check the distribution of spam and ham 
df.CLASS.value_counts()
# There is an almost equal distribution. Given that this is a small data set, this is probably good, 
# because the algorithm has enough items it can learn from
# Now, let us set up our X and y
X = df.CONTENT
y = df.CLASS
Explanation: <a id='section3c'></a>
Data cleanup
End of explanation
# Let us now split the data set into train and test sets
# We will use an 80/20 split
test_size = 0.2
seed = 42
scoring = 'accuracy'
num_folds = 10
from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=seed, test_size=test_size)
from sklearn.linear_model import LogisticRegression
from sklearn.naive_bayes import MultinomialNB
from sklearn.neighbors import KNeighborsClassifier
from sklearn.svm import SVC
from sklearn.tree import DecisionTreeClassifier
models = []
names = []
results = []
lr = ('LR', LogisticRegression())
knn = ('KNN', KNeighborsClassifier())
svc = ('SVC', SVC())
nb = ('NB', MultinomialNB())
cart = ('CART', DecisionTreeClassifier())
models.extend([lr,  knn, svc, nb, cart])
Explanation: <a id='section3d'></a>
Split the data
End of explanation
# Set up a vectorizer, and create a Document-Term matrix
from sklearn.feature_extraction.text import CountVectorizer
vect = CountVectorizer()
X_train_dtm = vect.fit_transform(X_train)
# Check the layout of the Document-Term matrix
X_train_dtm
Explanation: <a id='section3e'></a>
Transform the data
End of explanation
from sklearn.model_selection import KFold, cross_val_score
for name, model in models:
    kfold = KFold(n_splits=num_folds, random_state=seed)
    score = cross_val_score(model, X_train_dtm, y_train, scoring=scoring, cv=kfold)
    names.append(name)
    results.append(score)
from sklearn.ensemble import AdaBoostClassifier, GradientBoostingClassifier, \
    RandomForestClassifier, ExtraTreesClassifier
ensembles = []
ensemble_names = []
ensemble_results = []
ensembles.append(('AB', AdaBoostClassifier()))
ensembles.append(('RF', RandomForestClassifier()))
ensembles.append(('ET', ExtraTreesClassifier()))
for name, model in ensembles:
    kfold = KFold(n_splits=num_folds, random_state=seed)
    score = cross_val_score(model, X_train_dtm, y_train, cv=kfold, scoring=scoring)
    ensemble_names.append(name)
    ensemble_results.append(score)
models_list = []
for i, name in enumerate(names):
    d = {'model': name, 'mean': results[i].mean(), 'std': results[i].std()}
    models_list.append(d)
for i, name in enumerate(ensemble_names):
    d = {'model': name, 'mean': results[i].mean(), 'std': results[i].std()}
    models_list.append(d)
    
models_df = pd.DataFrame(models_list).set_index('model')
models_df.sort_values('mean', ascending=False)
Explanation: <a id='section3f'></a>
Build the model
In this step, we will build 6 models, and pick the one with the best accuracy score
End of explanation
cart
from sklearn.model_selection import GridSearchCV
final_model = DecisionTreeClassifier()
criterion_values = ['gini', 'entropy']
splitter_values = ['best', 'random']
min_samples_split_values = np.arange(2, 11, 1)
param_grid = dict(criterion=criterion_values, splitter=splitter_values, 
                  min_samples_split=min_samples_split_values)
kfold = KFold(n_splits=num_folds, random_state=seed)
grid = GridSearchCV(estimator=final_model, cv=kfold, scoring=scoring, param_grid=param_grid)
grid_result = grid.fit(X_train_dtm, y_train)
print(grid_result.best_params_, grid_result.best_score_)
Explanation: Model selection
Based on accuracy scores, the best algorithm is the Decision Tree Classifier. Logistic Regression and AdaBoost Classifier also performed very well. We will choose Decision Tree as our model, and look to tune it.
End of explanation
final_model = DecisionTreeClassifier(min_samples_split=7, random_state=seed)
final_model.fit(X_train_dtm, y_train)
# Transform the test data to a DTM and predict
X_test_dtm = vect.transform(X_test)
y_pred = final_model.predict(X_test_dtm)
Explanation: It looks like we were able to eke out some improvement in the performance. The Decision Tree Classifier seems to perform best with the min_samples_split set to 7. We will use this for our final model. Note that the default values for 'criterion' and 'splitter' seem to be part of the best performing set of parameters.
<a id='section3g'></a>
Run the prediction
End of explanation
# Let us check the accuracy score
# It needs to better than 50%, which was the baseline
from sklearn.metrics import accuracy_score, confusion_matrix, roc_auc_score
accuracy_score(y_test, y_pred)
# The accuracy score was 93.37%, which is lower than we may have anticipated 
# Let us check the confusion matrix to get a sense of the prediction distribution
confusion_matrix(y_test, y_pred)
# The model predicted 366 out of 392 instances correctly
# We had 14 false positives and 12 false negatives
# What were the false positive comments? (That is, ham marked as spam)
X_test[y_pred > y_test]
# And what were the false negative comments? (That is, spam comments that went undetected)
X_test[y_pred < y_test]
Explanation: <a id='section3h'></a>
Score the prediction
End of explanation
roc_auc_score(y_test, final_model.predict_proba(X_test_dtm)[:, 1])
Explanation: Some of the false negatives seem like they should have been marked as spam, so it is interesting that the model missed these. We may need to tune our vectorizer and/or attempt some other classifiers.
Let us check the area under the ROC curve.
End of explanation
<END_TASK>