<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 Frame
The client bank XYZ is running a direct marketing campaign. It wants to identify customers who would potentially be buying their new term deposit plan.
Acquire
Data is obtained from UCI Machine Learning repository. 
http
Step1:  Exercise 1
print the number of rows and columns of train and test
Exercise 2
Print the first 10 rows of train
Exercise 3
Print the column types of train and test. Are they the same in both train and test?
Step2:  Exercise 4
Find if any column has missing value
There is a pd.isnull function. How to use that?
Step3:  Exercise 5
Find % of customers in the input dataset who have purchased the term deposit
Step4:  Exercise 6
Did it drop? If not, what has to be done?
Exercise 7
Print columnn names of input
Step5:  Exercise 8
Create inputInteger and inputCategorical - two datasets - one having integer variables and another having categorical variables
Step6:  Exercise 9
Find length of categorical_variables
Step7:  Exercise 10
Convert inputInteger to numpy array
Step8:  Exercise 11
Now, create the inputUpdated array that has both inputInteger and inputCategorical concatenated
Hint Check function called vstack and hstack
Step9:  Train the model
Model 1
Step10:  Exercise 12
Now, change the max_depth = 6 and check the results.
Then, change the max_depth= None and check the results
Step11:  Exercise 13
Instead of predicting classes directly, predict the probability and check the auc
Step12:  Accuracy Metrics
AUC
ROC
Misclassification Rate
Confusion Matrix
Precision & Recall
Confusion Matrix
<img src="img/confusion_matrix.jpg" style="width
Step13:  Ensemble Trees
<img src="img/tree_ensemble1.png" style="width
Step14:  Exercise 14
Do the following
Predict on test
Find accuracy metrics
Step15:  Another way of encoding
One Hot Encoding
<img src="img/onehot.png" style="width
<ASSISTANT_TASK:>
Python Code:
#Import the necessary libraries
import numpy as np
import pandas as pd
#Read the train and test data
train = pd.read_csv("../data/train.csv")
test = pd.read_csv("../data/test.csv")
Explanation: Frame
The client bank XYZ is running a direct marketing campaign. It wants to identify customers who would potentially be buying their new term deposit plan.
Acquire
Data is obtained from UCI Machine Learning repository. 
http://mlr.cs.umass.edu/ml/datasets/Bank+Marketing
Data from direct marketing campaign (phone calls) of a Portuguese Bank is provided. 
Attribute Information:
bank client data:
age (numeric)
job : type of job (categorical: 'admin.','blue-collar','entrepreneur','housemaid','management','retired','self-employed','services','student','technician','unemployed','unknown')
marital : marital status (categorical: 'divorced','married','single','unknown'; note: 'divorced' means divorced or widowed)
education (categorical: 'basic.4y','basic.6y','basic.9y','high.school','illiterate','professional.course','university.degree','unknown')
default: has credit in default? (categorical: 'no','yes','unknown')
housing: has housing loan? (categorical: 'no','yes','unknown')
loan: has personal loan? (categorical: 'no','yes','unknown')
related with the last contact of the current campaign:
contact: contact communication type (categorical: 'cellular','telephone') 
month: last contact month of year (categorical: 'jan', 'feb', 'mar', ..., 'nov', 'dec')
day_of_week: last contact day of the week (categorical: 'mon','tue','wed','thu','fri')
duration: last contact duration, in seconds (numeric). Important note: this attribute highly affects the output target (e.g., if duration=0 then y='no'). Yet, the duration is not known before a call is performed. Also, after the end of the call y is obviously known. Thus, this input should only be included for benchmark purposes and should be discarded if the intention is to have a realistic predictive model.
other attributes:
campaign: number of contacts performed during this campaign and for this client (numeric, includes last contact)
pdays: number of days that passed by after the client was last contacted from a previous campaign (numeric; 999 means client was not previously contacted)
previous: number of contacts performed before this campaign and for this client (numeric)
poutcome: outcome of the previous marketing campaign (categorical: 'failure','nonexistent','success')
social and economic context attributes
emp.var.rate: employment variation rate - quarterly indicator (numeric)
cons.price.idx: consumer price index - monthly indicator (numeric) 
cons.conf.idx: consumer confidence index - monthly indicator (numeric) 
euribor3m: euribor 3 month rate - daily indicator (numeric)
nr.employed: number of employees - quarterly indicator (numeric)
Output variable (desired target):
y - has the client subscribed a term deposit? (binary: 'yes','no')
The given data is randomly divided into train and test for the purpose of this workshop. Build the model for train and use it to predict on test. 
Explore
End of explanation
#train
#test
#Are they the same?
#Combine train and test
frames = [train, test]
input = pd.concat(frames)
#Print first 10 records of input
Explanation: Exercise 1
print the number of rows and columns of train and test
Exercise 2
Print the first 10 rows of train
Exercise 3
Print the column types of train and test. Are they the same in both train and test?
End of explanation
#Replace deposit with a numeric column
#First, set all labels to be 0
input.at[:, "depositLabel"] = 0
#Now, set depositLabel to 1 whenever deposit is yes
input.at[input.deposit=="yes", "depositLabel"] = 1
Explanation: Exercise 4
Find if any column has missing value
There is a pd.isnull function. How to use that?
End of explanation
#Create the labels 
labels = 
labels
#Drop the deposit column 
input.drop(["deposit", "depositLabel"], axis=1)
Explanation: Exercise 5
Find % of customers in the input dataset who have purchased the term deposit
End of explanation
#Get list of columns that are continuous/integer
continuous_variables = input.dtypes[input.dtypes != "object"].index
continuous_variables
#Get list of columns that are categorical
categorical_variables = input.dtypes[input.dtypes=="object"].index
categorical_variables
Explanation: Exercise 6
Did it drop? If not, what has to be done?
Exercise 7
Print columnn names of input
End of explanation
inputInteger = 
#print inputInteger
inputInteger.head()
inputCategorical = 
#print inputCategorical
inputCategorical.head()
#Convert categorical variables into Labels using labelEncoder
inputCategorical = np.array(inputCategorical)
Explanation: Exercise 8
Create inputInteger and inputCategorical - two datasets - one having integer variables and another having categorical variables
End of explanation
#Load the preprocessing module
from sklearn import preprocessing
for i in range(len(categorical_variables)):
    lbl = preprocessing.LabelEncoder()
    lbl.fit(list(inputCategorical[:,i]))
    inputCategorical[:, i] = lbl.transform(inputCategorical[:, i])
#print inputCategorical
Explanation: Exercise 9
Find length of categorical_variables
End of explanation
inputInteger = 
inputInteger
Explanation: Exercise 10
Convert inputInteger to numpy array
End of explanation
inputUpdated.shape
Explanation: Exercise 11
Now, create the inputUpdated array that has both inputInteger and inputCategorical concatenated
Hint Check function called vstack and hstack
End of explanation
from sklearn import tree
from sklearn.externals.six import StringIO
import pydot
bankModelDT = tree.DecisionTreeClassifier(max_depth=2)
bankModelDT.fit(inputUpdated[:train.shape[0],:], labels[:train.shape[0]])
dot_data = StringIO() 
tree.export_graphviz(bankModelDT, out_file=dot_data) 
graph = pydot.graph_from_dot_data(dot_data.getvalue()) 
graph.write_pdf("bankDT.pdf") 
#Check the pdf
Explanation: Train the model
Model 1: Decision Tree
End of explanation
# Prediction
prediction_DT = bankModelDT.predict(inputUpdated[train.shape[0]:,:])
#Compute the error metrics
import sklearn.metrics
sklearn.metrics.auc(labels[train.shape[0]:], prediction_DT)
#What does that tell?
#What's the error AUC for the other Decision Tree Models
Explanation: Exercise 12
Now, change the max_depth = 6 and check the results.
Then, change the max_depth= None and check the results
End of explanation
sklearn.metrics.auc(labels[train.shape[0]:], prediction_DT[:,0])
Explanation: Exercise 13
Instead of predicting classes directly, predict the probability and check the auc
End of explanation
#Precision and Recall
sklearn.metrics.precision_score(labels[train.shape[0]:], prediction_DT)
sklearn.metrics.recall_score(labels[train.shape[0]:], prediction_DT)
Explanation: Accuracy Metrics
AUC
ROC
Misclassification Rate
Confusion Matrix
Precision & Recall
Confusion Matrix
<img src="img/confusion_matrix.jpg" style="width:604px;height:428px;">
Calculate True Positive Rate
TPR = TP / (TP+FN)
Calculate False Positive Rate
FPR = FP / (FP+TN)
Precision
Recall
End of explanation
from sklearn.ensemble import RandomForestClassifier
bankModelRF = RandomForestClassifier(n_jobs=-1, oob_score=True)
bankModelRF.fit(inputUpdated[:train.shape[0],:], labels[:train.shape[0]])
bankModelRF.oob_score_
Explanation: Ensemble Trees
<img src="img/tree_ensemble1.png" style="width:604px;height:428px;">
<br>
<br>
<br>
<br>
<br>
<br>
<img src="img/tree_ensemble2.png" style="width:604px;height:428px;">
src: http://www.slideshare.net/hustwj/scaling-up-machine-learning-the-tutorial-kdd-2011-part-iia-tree-ensembles
Random Forest
<img src="img/random_forest.jpg" style="width:604px;height:428px;">
src: http://www.slideshare.net/0xdata/jan-vitek-distributedrandomforest522013
End of explanation
import xgboost as xgb
params = {}
params["min_child_weight"] = 3
params["subsample"] = 0.7
params["colsample_bytree"] = 0.7
params["scale_pos_weight"] = 1
params["silent"] = 0
params["max_depth"] = 4
params["nthread"] = 6
params["gamma"] = 1
params["objective"] = "binary:logistic"
params["eta"] = 0.005
params["base_score"] = 0.1
params["eval_metric"] = "auc"
params["seed"] = 123
plst = list(params.items())
num_rounds = 120
xgtrain_pv = xgb.DMatrix(inputUpdated[:train.shape[0],:], label=labels[:train.shape[0]])
watchlist = [(xgtrain_pv, 'train')]
bankModelXGB = xgb.train(plst, xgtrain_pv, num_rounds)
prediction_XGB = bankModelXGB.predict(xgb.DMatrix(inputUpdated[train.shape[0]:,:]))
sklearn.metrics.auc(labels[train.shape[0]:], prediction_XGB)
Explanation: Exercise 14
Do the following
Predict on test
Find accuracy metrics: AUC, Precision, Recall 
How does it compare against Decision Tree
Gradient Boosting Machines
<img src="img/boosting.jpg" style="width:604px;height:428px;">
src: http://www.slideshare.net/hustwj/scaling-up-machine-learning-the-tutorial-kdd-2011-part-iia-tree-ensembles
End of explanation
inputOneHot = pd.get_dummies(input)
Explanation: Another way of encoding
One Hot Encoding
<img src="img/onehot.png" style="width:404px;height:228px;">
Whiteboard !
End of explanation
<END_TASK>