<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 Applying Machine Learning Techniques-Regression
Homepage
Step1:  Load Corpus After Preprocessing ...
Step2:  Basic & Advanced machine learning tools
Agenda
What is machine learning?
What are the two main categories of machine learning?
What are some examples of machine learning?
How does machine learning "work"?
What is machine learning?
One definition
Step4:  What are the features?
- AgeClass
Step5:  Classification accuracy
Step6:  Null accuracy
Step7:  Comparing the true and predicted response values
Step8:  Conclusion
Step9:  Basic terminology
True Positives (TP)
Step10:  Metrics computed from a confusion matrix
Classification Accuracy
Step11:  Classification Error
Step12:  Specificity
Step13:  False Positive Rate
Step14:  Precision
Step15:  Many other metrics can be computed
Step16:  Classifier comparison
http
Step17:  Decision Tree Regressor
Ref
Step18:  Random Forests
Naive Bayes
Simple Linear Regression
Recall that Simple Linear Regression is given by the following equation
Step19:  Multiple Linear Regresssion
Formally, multiple linear regression is the following model
Step20:  Polynomialy Regression
Quadratic Regression, regession with a second order polynomial, is given by the following formula
<ASSISTANT_TASK:>
Python Code:
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
%matplotlib inline
import seaborn as sns
import random
Explanation: Applying Machine Learning Techniques-Regression
Homepage: https://github.com/tien-le/kaggle-titanic
Updating later ...
End of explanation
#Training Corpus
trn_corpus_after_preprocessing = pd.read_csv("output/trn_corpus_after_preprocessing.csv")
#Testing Corpus
tst_corpus_after_preprocessing = pd.read_csv("output/tst_corpus_after_preprocessing.csv")
#tst_corpus_after_preprocessing[tst_corpus_after_preprocessing["Fare"].isnull()]
trn_corpus_after_preprocessing.info()
print("-"*36)
tst_corpus_after_preprocessing.info()
Explanation: Load Corpus After Preprocessing ...
End of explanation
trn_corpus_after_preprocessing.columns
list_of_non_preditor_variables = ['Survived','PassengerId']
#Method 1
#x_train = trn_corpus_after_preprocessing.ix[:, trn_corpus_after_preprocessing.columns != 'Survived']
#y_train = trn_corpus_after_preprocessing.ix[:,"Survived"]
#Method 2
x_train = trn_corpus_after_preprocessing[trn_corpus_after_preprocessing.columns.difference(list_of_non_preditor_variables)].copy()
y_train = trn_corpus_after_preprocessing['Survived'].copy()
#y_train = trn_corpus_after_preprocessing.iloc[:,-1]
#y_train = trn_corpus_after_preprocessing[trn_corpus_after_preprocessing.columns[-1]]
#x_train
#y_train
x_train.columns
# check the types of the features and response
#print(type(x_train))
#print(type(x_test))
#Method 1
#x_test = tst_corpus_after_preprocessing.ix[:, trn_corpus_after_preprocessing.columns != 'Survived']
#y_test = tst_corpus_after_preprocessing.ix[:,"Survived"]
#Method 2
x_test = tst_corpus_after_preprocessing[tst_corpus_after_preprocessing.columns.difference(list_of_non_preditor_variables)].copy()
y_test = tst_corpus_after_preprocessing['Survived'].copy()
#y_test = tst_corpus_after_preprocessing.iloc[:,-1]
#y_test = tst_corpus_after_preprocessing[tst_corpus_after_preprocessing.columns[-1]]
#x_test
#y_test
# display the first 5 rows
x_train.head()
# display the last 5 rows
x_train.tail()
# check the shape of the DataFrame (rows, columns)
x_train.shape
Explanation: Basic & Advanced machine learning tools
Agenda
What is machine learning?
What are the two main categories of machine learning?
What are some examples of machine learning?
How does machine learning "work"?
What is machine learning?
One definition: "Machine learning is the semi-automated extraction of knowledge from data"
Knowledge from data: Starts with a question that might be answerable using data
Automated extraction: A computer provides the insight
Semi-automated: Requires many smart decisions by a human
What are the two main categories of machine learning?
Supervised learning: Making predictions using data
Example: Is a given email "spam" or "ham"?
There is an outcome we are trying to predict
Unsupervised learning: Extracting structure from data
Example: Segment grocery store shoppers into clusters that exhibit similar behaviors
There is no "right answer"
How does machine learning "work"?
High-level steps of supervised learning:
First, train a machine learning model using labeled data
"Labeled data" has been labeled with the outcome
"Machine learning model" learns the relationship between the attributes of the data and its outcome
Then, make predictions on new data for which the label is unknown
The primary goal of supervised learning is to build a model that "generalizes": It accurately predicts the future rather than the past!
Questions about machine learning
How do I choose which attributes of my data to include in the model?
How do I choose which model to use?
How do I optimize this model for best performance?
How do I ensure that I'm building a model that will generalize to unseen data?
Can I estimate how well my model is likely to perform on unseen data?
Benefits and drawbacks of scikit-learn
Benefits:
Consistent interface to machine learning models
Provides many tuning parameters but with sensible defaults
Exceptional documentation
Rich set of functionality for companion tasks
Active community for development and support
Potential drawbacks:
Harder (than R) to get started with machine learning
Less emphasis (than R) on model interpretability
Further reading:
Ben Lorica: Six reasons why I recommend scikit-learn
scikit-learn authors: API design for machine learning software
Data School: Should you teach Python or R for data science?
Types of supervised learning
Classification: Predict a categorical response
Regression: Predict a ordered/continuous response
Note that each value we are predicting is the response (also known as: target, outcome, label, dependent variable)
Model evaluation metrics
Regression problems: Mean Absolute Error, Mean Squared Error, Root Mean Squared Error
Classification problems: Classification accuracy
Load Corpus
End of explanation
from sklearn import tree
clf = tree.DecisionTreeClassifier()
clf = clf.fit(x_train, y_train)
#Once trained, we can export the tree in Graphviz format using the export_graphviz exporter. 
#Below is an example export of a tree trained on the entire iris dataset:
with open("output/titanic.dot", 'w') as f:
    f = tree.export_graphviz(clf, out_file=f)
#Then we can use Graphviz’s dot tool to create a PDF file (or any other supported file type): 
#dot -Tpdf titanic.dot -o titanic.pdf.
import os
os.unlink('output/titanic.dot')
#Alternatively, if we have Python module pydotplus installed, we can generate a PDF file 
#(or any other supported file type) directly in Python:
import pydotplus 
dot_data = tree.export_graphviz(clf, out_file=None) 
graph = pydotplus.graph_from_dot_data(dot_data) 
graph.write_pdf("output/titanic.pdf")
#The export_graphviz exporter also supports a variety of aesthetic options, 
#including coloring nodes by their class (or value for regression) 
#and using explicit variable and class names if desired. 
#IPython notebooks can also render these plots inline using the Image() function:
from IPython.display import Image  
dot_data = tree.export_graphviz(clf, out_file=None, 
                         feature_names= list(x_train.columns[1:]), #iris.feature_names,  
                         class_names= ["Survived"], #iris.target_names,  
                         filled=True, rounded=True,  
                         special_characters=True)  
graph = pydotplus.graph_from_dot_data(dot_data)  
Image(graph.create_png())
print("accuracy score: ", clf.score(x_test,y_test))
Explanation: What are the features?
- AgeClass: 
- AgeClassSquared: 
- AgeSquared: 
- ...
What is the response?
- Survived: 1-Yes, 0-No
What else do we know?
- Because the response variable is dicrete, this is a Classification problem.
- There are 200 observations (represented by the rows), and each observation is a single market.
Note that if the response variable is continuous, this is a regression problem.
Decision Trees Classification
End of explanation
#After being fitted, the model can then be used to predict the class of samples:
y_pred_class = clf.predict(x_test);
#Alternatively, the probability of each class can be predicted, 
#which is the fraction of training samples of the same class in a leaf:
clf.predict_proba(x_test);
# calculate accuracy
from sklearn import metrics
print(metrics.accuracy_score(y_test, y_pred_class))
Explanation: Classification accuracy: percentage of correct predictions
End of explanation
# examine the class distribution of the testing set (using a Pandas Series method)
y_test.value_counts()
# calculate the percentage of ones
y_test.mean()
# calculate the percentage of zeros
1 - y_test.mean()
# calculate null accuracy (for binary classification problems coded as 0/1)
max(y_test.mean(), 1 - y_test.mean())
# calculate null accuracy (for multi-class classification problems)
y_test.value_counts().head(1) / len(y_test)
Explanation: Null accuracy: accuracy that could be achieved by always predicting the most frequent class
End of explanation
# print the first 25 true and predicted responses
from __future__ import print_function
print('True:', y_test.values[0:25])
print('Pred:', y_pred_class[0:25])
Explanation: Comparing the true and predicted response values
End of explanation
# IMPORTANT: first argument is true values, second argument is predicted values
print(metrics.confusion_matrix(y_test, y_pred_class))
Explanation: Conclusion: ???
Classification accuracy is the easiest classification metric to understand
But, it does not tell you the underlying distribution of response values
And, it does not tell you what "types" of errors your classifier is making
Confusion matrix
Table that describes the performance of a classification model
End of explanation
# save confusion matrix and slice into four pieces
confusion = metrics.confusion_matrix(y_test, y_pred_class)
TP = confusion[1, 1]
TN = confusion[0, 0]
FP = confusion[0, 1]
FN = confusion[1, 0]
print(TP, TN, FP, FN)
Explanation: Basic terminology
True Positives (TP): we correctly predicted that they do have diabetes
True Negatives (TN): we correctly predicted that they don't have diabetes
False Positives (FP): we incorrectly predicted that they do have diabetes (a "Type I error")
False Negatives (FN): we incorrectly predicted that they don't have diabetes (a "Type II error")
End of explanation
print((TP + TN) / float(TP + TN + FP + FN))
print(metrics.accuracy_score(y_test, y_pred_class))
Explanation: Metrics computed from a confusion matrix
Classification Accuracy: Overall, how often is the classifier correct?
End of explanation
print((FP + FN) / float(TP + TN + FP + FN))
print(1 - metrics.accuracy_score(y_test, y_pred_class))
Explanation: Classification Error: Overall, how often is the classifier incorrect?
Also known as "Misclassification Rate"
End of explanation
print(TN / float(TN + FP))
Explanation: Specificity: When the actual value is negative, how often is the prediction correct?
How "specific" (or "selective") is the classifier in predicting positive instances?
End of explanation
print(FP / float(TN + FP))
Explanation: False Positive Rate: When the actual value is negative, how often is the prediction incorrect?
End of explanation
print(TP / float(TP + FP))
print(metrics.precision_score(y_test, y_pred_class))
print("Presicion: ", metrics.precision_score(y_test, y_pred_class))
print("Recall: ", metrics.recall_score(y_test, y_pred_class))
print("F1 score: ", metrics.f1_score(y_test, y_pred_class))
Explanation: Precision: When a positive value is predicted, how often is the prediction correct?
How "precise" is the classifier when predicting positive instances?
End of explanation
from sklearn import svm
model = svm.LinearSVC()
model.fit(x_train, y_train)
acc_score = model.score(x_test, y_test)
print("Accuracy score: ", acc_score)
y_pred_class = model.predict(x_test) 
from sklearn import metrics
confusion_matrix = metrics.confusion_matrix(y_test, y_pred_class)
print(confusion_matrix)
Explanation: Many other metrics can be computed: F1 score, Matthews correlation coefficient, etc.
Conclusion:
Confusion matrix gives you a more complete picture of how your classifier is performing
Also allows you to compute various classification metrics, and these metrics can guide your model selection
Which metrics should you focus on?
Choice of metric depends on your business objective
Spam filter (positive class is "spam"): Optimize for precision or specificity because false negatives (spam goes to the inbox) are more acceptable than false positives (non-spam is caught by the spam filter)
Fraudulent transaction detector (positive class is "fraud"): Optimize for sensitivity because false positives (normal transactions that are flagged as possible fraud) are more acceptable than false negatives (fraudulent transactions that are not detected)
Support Vector Machine (SVM)
Linear Support Vector Classification.
Similar to SVC with parameter kernel=’linear’, but implemented in terms of liblinear rather than libsvm, so it has more flexibility in the choice of penalties and loss functions and should scale better to large numbers of samples.
Ref: http://scikit-learn.org/stable/modules/generated/sklearn.svm.LinearSVC.html#sklearn.svm.LinearSVC
End of explanation
from sklearn.neighbors import KNeighborsClassifier
from sklearn.svm import SVC
from sklearn.gaussian_process import GaussianProcessClassifier
from sklearn.gaussian_process.kernels import RBF
from sklearn.tree import DecisionTreeClassifier
from sklearn.ensemble import RandomForestClassifier, AdaBoostClassifier
from sklearn.neural_network import MLPClassifier
from sklearn.naive_bayes import GaussianNB
from sklearn.discriminant_analysis import QuadraticDiscriminantAnalysis
from sklearn.datasets import make_classification
from sklearn.preprocessing import StandardScaler
from matplotlib.colors import ListedColormap
#classifiers
#x_train
#sns.pairplot(x_train)
x_train_scaled = StandardScaler().fit_transform(x_train)
x_test_scaled = StandardScaler().fit_transform(x_test)
x_train_scaled[0]
len(x_train_scaled[0])
df_x_train_scaled = pd.DataFrame(columns=x_train.columns, data=x_train_scaled)
df_x_train_scaled.head()
#sns.pairplot(df_x_train_scaled)
names = ["Nearest Neighbors", "Linear SVM", "RBF SVM",
         "Decision Tree", "Random Forest", "Neural Net", "AdaBoost",
         "Naive Bayes", "QDA", "Gaussian Process"]
classifiers = [
    KNeighborsClassifier(3),
    SVC(kernel="linear", C=0.025),
    SVC(gamma=2, C=1),
    DecisionTreeClassifier(max_depth=5),
    RandomForestClassifier(max_depth=5, n_estimators=10, max_features=1),
    MLPClassifier(alpha=1),
    AdaBoostClassifier(),
    GaussianNB(),
    QuadraticDiscriminantAnalysis()
    #, GaussianProcessClassifier(1.0 * RBF(1.0), warm_start=True), # Take too long...
    ]
# iterate over classifiers
for name, model in zip(names, classifiers):
    model.fit(x_train_scaled, y_train)
    acc_score = model.score(x_test_scaled, y_test)
    print(name, " - accuracy score: ", acc_score)
#end for
Explanation: Classifier comparison
http://scikit-learn.org/stable/auto_examples/classification/plot_classifier_comparison.html
A comparison of a several classifiers in scikit-learn on synthetic datasets. The point of this example is to illustrate the nature of decision boundaries of different classifiers. This should be taken with a grain of salt, as the intuition conveyed by these examples does not necessarily carry over to real datasets.
Particularly in high-dimensional spaces, data can more easily be separated linearly and the simplicity of classifiers such as naive Bayes and linear SVMs might lead to better generalization than is achieved by other classifiers.
The plots show training points in solid colors and testing points semi-transparent. The lower right shows the classification accuracy on the test set.
End of explanation
from sklearn import tree
clf = tree.DecisionTreeRegressor()
clf = clf.fit(x_train, y_train)
clf.score(x_test,y_test)
#clf.predict(x_test)
Explanation: Decision Tree Regressor
Ref: http://scikit-learn.org/stable/modules/tree.html
Decision trees can also be applied to regression problems, using the DecisionTreeRegressor class.
As in the classification setting, the fit method will take as argument arrays X and y, only that in this case y is expected to have floating point values instead of integer values:
End of explanation
from sklearn.linear_model import LinearRegression
model = LinearRegression()
model.fit(x_train, y_train)
r_squared = model.score(x_test, y_test)
print("R-squared: %.4f" %r_squared)
Explanation: Random Forests
Naive Bayes
Simple Linear Regression
Recall that Simple Linear Regression is given by the following equation:
$y = \alpha + \beta x$
Our goal is to solve the values $\alpha$ and $\beta$ that minimize the cost function. 
$$\beta = \frac{cov(x,y)}{var(x)}$$
where $cov(x,y)$ denotes a measure of how far a set of values is spread out. 
Note that: 
* Variance is zero if all of the features are spread out equally. 
* A SMALL variance indicates that the numbers are NEAR the mean of the set
* A LARGE variance when the numbers are FAR the mean of the set
$$var(x) = \frac{\sum\limits_{i=1}^{n}{\left( x_i - \overline{x} \right)}}{n-1}$$
$$cov(x,y) = \frac{\sum\limits_{i=1}^{n}{\left( x_i - \overline{x} \right)\left( y_i - \overline{y} \right)}}{n-1}$$
Having solved $\beta$, we can estimate $\alpha$ using the following formula:
$$\alpha = \overline{y} - \beta \overline{x}$$
Evaluating the Model
Using r-squared - that measures how well the observed values of the response variables are predicted by the model. In the case of simple linear regression, r-squared is equal to Pearson's r. In this method, r-squared must be  a positive number between zero and one. In others, r-squared can return a negative number if the model performs extremely poorly.
End of explanation
from sklearn.linear_model import LinearRegression
model = LinearRegression()
model.fit(x_train, y_train)
predictions = model.predict(x_test)
#for i in range(predictions.size):
#    print("Predicted: %.2f, Target: %.2f" %(predictions[i], y_test[i]))
r_squared = model.score(x_test, y_test)
    
print("R-squared: %.4f" %r_squared)
Explanation: Multiple Linear Regresssion
Formally, multiple linear regression is the following model:
$$y = \alpha+\beta_1x_1+\beta_2x_2+...+\beta_nx_n$$
or 
$$Y = X\beta$$
where $Y$ denotes a column vector of the values of the response variables for training, $\beta$ denotes a column vector of the values of the model's parameters, $X$ is called the design matrix, an $m \times n$ dimensional matrix of the values of the features. 
We can solve $\beta$ as follows:
$$\beta = \left( X^TX \right)^{-1}X^TY$$
Note that - code python:
python
from numpy import dot, transpose
beta = dot(inv(dot(transpose(X),X)), dot(transpose(X), Y))
End of explanation
import numpy as np
import matplotlib.pyplot as plt
from sklearn.linear_model import LinearRegression
from sklearn.preprocessing import PolynomialFeatures
model = LinearRegression()
model.fit(x_train, y_train)
xx = np.linspace(0, 26, 100)
#yy = np.linspace(0, 26, 100)
#yy = model.predict(xx.reshape(xx.shape[0],1))
#plt.plot(xx, yy)
quadratic_featurizer = PolynomialFeatures(degree=2)
x_train_quadratic = quadratic_featurizer.fit_transform(x_train)
x_test_quadratic = quadratic_featurizer.fit(x_test)
x_train.head()
model_quadratic = LinearRegression()
model_quadratic.fit(x_train_quadratic, y_train)
#predictions = model_quadratic.predict(x_test_quadratic)
#r_squared = model_quadratic.score(x_test_quadratic, y_test)
#r_squared
    
#print("R-squared: %.4f" %r_squared)
Explanation: Polynomialy Regression
Quadratic Regression, regession with a second order polynomial, is given by the following formula:
$$y = \alpha +\beta_1x^1+\beta_2x^2$$
End of explanation
<END_TASK>