<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 <h1>Machine learning using Regression</h1>
<h2>Read the data</h2>
<h2>Generate a few summary statistics</h2>
<h3>Data set 1
Step1:  <h4>See all columns</h4>
Step2:  <h4>Examine the distribution of the data in column 4</h4>
<li>Quartile 1
Step3:  <h4>Examine the dependent variable</h4>
Step4:  <h4>Examine correlations</h4>
Step5:  <h4>Highly correlated items = not good!</h4>
<h4>Low correlated items = good </h4>
<h4>Correlations with target (dv) = good (high predictive power)</h4>
<h3>Data Set 2
Step6:  <h3>Examining the correlation of one variable with the others</h3>
Step7:  <h3>Pandas scatter matrix function helps visualize the relationship between features</h3>
Use with care though, because it is processor intensive
Step8:  <h3>And we can examine quintile plots as we did with the rocks and mines data</h3>
Step9:  <h2>Training a classifier on Rocks vs Mines</h2>
Step10:  <h4>Convert labels R and M to 0 and 1</h4>
Step11:  <h4>Divide the dataset into training and test samples</h4>
<h4>Separate out the x and y variable frames for the train and test samples</h4>
Step12:  <h2>Build the model and fit the training data</h2>
Step13:  <h1>Interpreting categorical prediction results</h1>
<h3>Precision
<h3>Recall
<h3>True Positive Rate
<h3>False Positive Rate
<h3>Precision recall curve
<h3>ROC curve
<h3>F-Score
<h3>Area under PR curve
<h3>Area under ROC curve
<h4>Generate predictions in-sample error</h4>
Step14:  <h3>These are horrible!</h3>
<b>But do we really care?</b>
<li>Focus on the problem
<li>Do we need to recognize both rocks as well as mines correctly?
<li>How do we interpret the predicted y-values
Step15:  <h2>We want to predict categories
Step16:  <h3>Misclassification rate = (fp + fn)/number of cases</h3>
Step17:  <h3>Precision and Recall</h3>
Step18:  <h2>Confusion matrix (and hence precision, recall etc.) depend on the selected threshold</h2>
<h4>As the threshold changes, we will need to tradeoff precision and recall</h4>
Step19:  <h2>ROC
Step20:  <h3>Repeat for the holdout sample</h3>
Step21:  <h2>Drawing the ROC Curve</h2>
<h3>sklearn has a function roc_curve that does this for us</h3>
Step22:  <h4>In-sample ROC Curve</h4>
Step23:  <h4>Out-sample ROC curve</h4>
Step24:  <h2>So, what threshold should we actually use?</h2>
<h4>ROC curves and AUC give you a sense for how good your classifier is and how sensitive it is to changes in threshold</h4>
<h4>Too sensitive is not good</h4>
<h3>Example
Step25:  <h3>Example
<ASSISTANT_TASK:>
Python Code:
import pandas as pd
from pandas import DataFrame
url="https://archive.ics.uci.edu/ml/machine-learning-databases/undocumented/connectionist-bench/sonar/sonar.all-data"
df = pd.read_csv(url,header=None)
df.describe()
Explanation: <h1>Machine learning using Regression</h1>
<h2>Read the data</h2>
<h2>Generate a few summary statistics</h2>
<h3>Data set 1: Rocks vs. Mines</h3>
<li>Independent variables: sonar soundings at different frequencies
<li>Dependent variable (target): Rock or Mine
End of explanation
pd.options.display.max_columns=70
df.describe()
Explanation: <h4>See all columns</h4>
End of explanation
import numpy as np 
import pylab 
import scipy.stats as stats
import matplotlib
import matplotlib.pyplot as plt
matplotlib.style.use('ggplot')
%matplotlib inline
   
stats.probplot(df[4], dist="norm", plot=pylab)
pylab.show()
Explanation: <h4>Examine the distribution of the data in column 4</h4>
<li>Quartile 1: from .0067 to .03805
<li>Quartile 2: from .03805 to .0625
<li>Quartile 3: from .0625 to .100275
<li>Quartile 4: from .100275 to .401
<h4>Quartile 4 is much larger than the other quartiles. This raises the possibility of outliers</h4>
<h4> A Quantile - Quantile (qq) plot can help identify outliers</h4>
<li>y-axis contains values
<li>x-axis is the cumulative normal density function plotted as a straight line (-3 to +3)
<li>y-axis is the values ordered from lowest to highest
<li>the closer the curve is to the line, the more it reflects a normal distribution
End of explanation
df[60].unique()
Explanation: <h4>Examine the dependent variable</h4>
End of explanation
df.corr()
import matplotlib.pyplot as plot
plot.pcolor(df.corr())
plot.show()
df.corr()[0].plot()
Explanation: <h4>Examine correlations</h4>
End of explanation
url = "http://archive.ics.uci.edu/ml/machine-learning-databases/wine-quality/winequality-red.csv"
import pandas as pd
from pandas import DataFrame
w_df = pd.read_csv(url,header=0,sep=';')
w_df.describe()
w_df['volatile acidity']
w_df.corr()
import matplotlib.pyplot as plot
plot.pcolor(w_df.corr())
plot.show()
Explanation: <h4>Highly correlated items = not good!</h4>
<h4>Low correlated items = good </h4>
<h4>Correlations with target (dv) = good (high predictive power)</h4>
<h3>Data Set 2: Wine data</h3>
<li>Independent variables: Wine composition (alcohol content, sulphites, acidity, etc.)
<li>Dependent variable (target): Taste score (average of a panel of 3 wine tasters)
End of explanation
w_df.corr()['fixed acidity'].plot()
Explanation: <h3>Examining the correlation of one variable with the others</h3>
End of explanation
from pandas.tools.plotting import scatter_matrix
p=scatter_matrix(w_df, alpha=0.2, figsize=(12, 12), diagonal='kde')
Explanation: <h3>Pandas scatter matrix function helps visualize the relationship between features</h3>
Use with care though, because it is processor intensive
End of explanation
import numpy as np 
import pylab 
import scipy.stats as stats
%matplotlib inline
   
stats.probplot(w_df['alcohol'], dist="norm", plot=pylab)
pylab.show()
Explanation: <h3>And we can examine quintile plots as we did with the rocks and mines data</h3>
End of explanation
import numpy
import random
from sklearn import datasets, linear_model
from sklearn.metrics import roc_curve, auc
import pylab as pl
import pandas as pd
from pandas import DataFrame
url="https://archive.ics.uci.edu/ml/machine-learning-databases/undocumented/connectionist-bench/sonar/sonar.all-data"
df = pd.read_csv(url,header=None)
df.describe()
Explanation: <h2>Training a classifier on Rocks vs Mines</h2>
End of explanation
df[60]=np.where(df[60]=='R',0,1)
Explanation: <h4>Convert labels R and M to 0 and 1</h4>
End of explanation
from sklearn.model_selection import train_test_split
train, test = train_test_split(df, test_size = 0.3)
x_train = train.iloc[0:,0:60]
y_train = train[60]
x_test = test.iloc[0:,0:60]
y_test = test[60]
y_train
Explanation: <h4>Divide the dataset into training and test samples</h4>
<h4>Separate out the x and y variable frames for the train and test samples</h4>
End of explanation
model = linear_model.LinearRegression()
model.fit(x_train,y_train)
Explanation: <h2>Build the model and fit the training data</h2>
End of explanation
training_predictions = model.predict(x_train)
print(np.mean((training_predictions - y_train) ** 2))
print('Train R-Square:',model.score(x_train,y_train))
print('Test R-Square:',model.score(x_test,y_test))
Explanation: <h1>Interpreting categorical prediction results</h1>
<h3>Precision
<h3>Recall
<h3>True Positive Rate
<h3>False Positive Rate
<h3>Precision recall curve
<h3>ROC curve
<h3>F-Score
<h3>Area under PR curve
<h3>Area under ROC curve
<h4>Generate predictions in-sample error</h4>
End of explanation
print(max(training_predictions),min(training_predictions),np.mean(training_predictions))
Explanation: <h3>These are horrible!</h3>
<b>But do we really care?</b>
<li>Focus on the problem
<li>Do we need to recognize both rocks as well as mines correctly?
<li>How do we interpret the predicted y-values
End of explanation
def confusion_matrix(predicted, actual, threshold):
    if len(predicted) != len(actual): return -1
    tp = 0.0
    fp = 0.0
    tn = 0.0
    fn = 0.0
    for i in range(len(actual)):
        if actual[i] > 0.5: #labels that are 1.0  (positive examples)
            if predicted[i] > threshold:
                tp += 1.0 #correctly predicted positive
            else:
                fn += 1.0 #incorrectly predicted negative
        else:              #labels that are 0.0 (negative examples)
            if predicted[i] < threshold:
                tn += 1.0 #correctly predicted negative
            else:
                fp += 1.0 #incorrectly predicted positive
    rtn = [tp, fn, fp, tn]
    return rtn
testing_predictions = model.predict(x_test)
testing_predictions = model.predict(x_test)
confusion_matrix(testing_predictions,np.array(y_test),0.5)
Explanation: <h2>We want to predict categories: Rocks or Mines</h2>
<h2>But we're actually getting a continuous value</h2>
<h2>Not the same thing. So R-Square probably doesn't mean a whole lot</h2>
<h2>We need to convert the conitnuous values into  categorical 1s and 0s. We can do this by fixing a threshold value between 0 and 1</h2>
<h2>Values greater than the threshold are 1 (Mines). Values less than or equal to the threshold are 0 (Rocks)
<h2>Confusion matrix</h2>
<li>Reports the proportion of
<ol>
<li><b>true positive</b>: predicts mine and is a mine
<li><b>false positive</b>: predicts mine and is not a mine
<li><b>true negative</b>: predicts not mine and is not a mine
<li><b>false negative</b>:Predicts not mine but turns out to be a mine (BOOM!)
End of explanation
cm = confusion_matrix(testing_predictions,np.array(y_test),0.5)
misclassification_rate = (cm[1] + cm[2])/len(y_test)
misclassification_rate
Explanation: <h3>Misclassification rate = (fp + fn)/number of cases</h3>
End of explanation
[tp, fn, fp, tn] = confusion_matrix(testing_predictions,np.array(y_test),0.5)
precision = tp/(tp+fp)
recall = tp/(tp+fn)
f_score = 2 * (precision * recall)/(precision + recall)
print(precision,recall,f_score)
Explanation: <h3>Precision and Recall</h3>
End of explanation
[tp, fn, fp, tn] = confusion_matrix(testing_predictions,np.array(y_test),0.9)
precision = tp/(tp+fp)
recall = tp/(tp+fn)
f_score = 2 * (precision * recall)/(precision + recall)
print(precision,recall,f_score)
Explanation: <h2>Confusion matrix (and hence precision, recall etc.) depend on the selected threshold</h2>
<h4>As the threshold changes, we will need to tradeoff precision and recall</h4>
End of explanation
positives = list()
negatives = list()
actual = np.array(y_train)
for i in range(len(y_train)):
    
    if actual[i]:
        positives.append(training_predictions[i])
    else:
        negatives.append(training_predictions[i])
df_p = pd.DataFrame(positives)
df_n = pd.DataFrame(negatives)
fig, ax = plt.subplots()
a_heights, a_bins = np.histogram(df_p)
b_heights, b_bins = np.histogram(df_n, bins=a_bins)
width = (a_bins[1] - a_bins[0])/3
ax.bar(a_bins[:-1], a_heights, width=width, facecolor='cornflowerblue')
ax.bar(b_bins[:-1]+width, b_heights, width=width, facecolor='seagreen')
Explanation: <h2>ROC: Receiver Order Characteristic</h2>
<li>An ROC curve shows the performance of a binary classifier as the threshold varies.
<li>Computes two series:
<ol>
<li>False positive rate (FPR) Fall out/false alarm  = False Positives/(True Negatives + False Positives)
<ul><li>Or, what proportion of rocks  are identified as mines</ul>
<li>True Positive rate (TPR) Sensitivity/recall = True Positives/(True Positives + False Negatives)
<ul><li>Or, what proportion of actual mines are identified as mines</ul>
</ol>
<ul>
<li><b>true positive</b>: predicts mine and is a mine
<li><b>false positive</b>: predicts mine and is not a mine
<li><b>true negative</b>: predicts not mine and is not a mine
<li><b>false negative</b>:Predicts not mine but turns out to be a mine (BOOM!)
<h3>Let's first plot the predictions against actuals<h3>
The goal is to see if our classifier has discriminated at all
End of explanation
positives = list()
negatives = list()
actual = np.array(y_test)
for i in range(len(y_test)):
    
    if actual[i]:
        positives.append(testing_predictions[i])
    else:
        negatives.append(testing_predictions[i])
df_p = pd.DataFrame(positives)
df_n = pd.DataFrame(negatives)
fig, ax = plt.subplots()
a_heights, a_bins = np.histogram(df_p)
b_heights, b_bins = np.histogram(df_n, bins=a_bins)
width = (a_bins[1] - a_bins[0])/3
ax.bar(a_bins[:-1], a_heights, width=width, facecolor='cornflowerblue')
ax.bar(b_bins[:-1]+width, b_heights, width=width, facecolor='seagreen')
Explanation: <h3>Repeat for the holdout sample</h3>
End of explanation
from sklearn.metrics import roc_curve, auc
Explanation: <h2>Drawing the ROC Curve</h2>
<h3>sklearn has a function roc_curve that does this for us</h3>
End of explanation
(fpr, tpr, thresholds) = roc_curve(y_train,training_predictions)
area = auc(fpr,tpr)
pl.clf() #Clear the current figure
pl.plot(fpr,tpr,label="In-Sample ROC Curve with area = %1.2f"%area)
pl.plot([0, 1], [0, 1], 'k') #This plots the random (equal probability line)
pl.xlim([0.0, 1.0])
pl.ylim([0.0, 1.0])
pl.xlabel('False Positive Rate')
pl.ylabel('True Positive Rate')
pl.title('In sample ROC rocks versus mines')
pl.legend(loc="lower right")
pl.show()
Explanation: <h4>In-sample ROC Curve</h4>
End of explanation
(fpr, tpr, thresholds) = roc_curve(y_test,testing_predictions)
area = auc(fpr,tpr)
pl.clf() #Clear the current figure
pl.plot(fpr,tpr,label="Out-Sample ROC Curve with area = %1.2f"%area)
pl.plot([0, 1], [0, 1], 'k')
pl.xlim([0.0, 1.0])
pl.ylim([0.0, 1.0])
pl.xlabel('False Positive Rate')
pl.ylabel('True Positive Rate')
pl.title('Out sample ROC rocks versus mines')
pl.legend(loc="lower right")
pl.show()
(fpr, tpr, thresholds)
Explanation: <h4>Out-sample ROC curve</h4>
End of explanation
cm = confusion_matrix(testing_predictions,np.array(y_test),.1)
cost1 = 1000*cm[0] + 300 * cm[2] + 200 * cm[1] + 200 * cm[3]
cm = confusion_matrix(testing_predictions,np.array(y_test),.9)
cost2 = 1000*cm[0] + 300 * cm[2] + 200 * cm[1] + 200 * cm[3]
print(cost1,cost2)
Explanation: <h2>So, what threshold should we actually use?</h2>
<h4>ROC curves and AUC give you a sense for how good your classifier is and how sensitive it is to changes in threshold</h4>
<h4>Too sensitive is not good</h4>
<h3>Example: Let's say</h3>
<li>Everything classified as a rock needs to be checked with a hand scanner at 200/scan</li>
<li>Everything classified as a mine needs to be defused at 1000 if it is a real mine or 300 if it turns out to be a rock</li>
End of explanation
cm = confusion_matrix(testing_predictions,np.array(y_test),.1)
cost1 = 0*cm[0] + 0 * cm[2] + 5000 * cm[1] + 0 * cm[3]
cm = confusion_matrix(testing_predictions,np.array(y_test),.9)
cost2 = 0*cm[0] + 0 * cm[2] + 5000 * cm[1] + 0 * cm[3]
print(cost1,cost2)
Explanation: <h3>Example: Let's say</h3>
<li>Everything classified as a rock will be assumed a rock and if wrong, will cost 5000 in injuries</li>
<li>Everything classified as a mine will be left as is (no one will walk on it!)</li>
End of explanation
<END_TASK>