<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 <table>
<tr>
<td width=15%><img src="./img/UGA.png"></img></td>
<td><center><h1>Introduction to Python for Data Sciences</h1></center></td>
<td width=15%><a href="http
Step1:  Support Vector Machines (SVM) are based on learning a vector $w$ and an intercept $b$ such that the hyperplane $w^T x - b = 0$ separates the data i.e.  $a$ belongs to one class if  $w^T a - b > 0$ and the other elsewhere. 
They were later extended to Kernel methods that is  $\kappa(w, a) - b = 0$ is now the separating curve where $\kappa$ is the kernel, typically
Step3:  The following illustration can be found in the Python Data Science Handbook by Jake VanderPlas.
Step4:  We see clearly that the linear SVM seeks at maximizing the margin between the hyperplane and the two well defined classes from the data.
Non-separable data
In real cases, the data is usually not linearly separable as before.
Step5:  Let us use the same linear SVM classifier. Obviously, there are misclassified points, the model is thus learnt not by maximizing the margin (which does not exist anymore) but by minimizing a penalty over misclassified data. This penalty takes the form of an allowance margin controlled by a parameter $C$. The smaller $C$ the more inclusive the margin. Finding a good value for $C$ is up to the data scientist.
Step6:  To find out which value of $C$ to use or globally the performance of the classifier, one can use Scikit Learn's classification metrics, for instance the confusion matrix.
Step7:  It can also be plotted in a fancier way with seaborn.
Step8:  Kernels
When the separation between classes is not linear, kernels may be used to draw separating curves instead of lines. The most popular is the Gaussian rbf.
Step9:  Let us compare the linear and rbf training error using the zero one loss (the proportion of misclassified examples).
Step10:  Multiple classes
Where there are multiples classes (as in the iris dataset of the Pandas notebook), different strategies can be adopted
Step11:  Other classifiers
The main classifiers from Scikit learn are
Step12:  One immediate problem here is that the features are not numeric (not floats). Thankfully, Scikit Learn provides encoders to convert categorical (aka nominal, discrete) features to numerical ones.
Step13:  Even numerical values were encoded, as we are going to normalize, it is not really important. 
The normalization is done by removing the mean and equalizing the variance per feature, in addition, we are going to add an intercept.
Step14:  Regression and Feature selection with the Lasso
The lasso problem is finding a regressor $w$ such that minimizes
$$  \frac{1}{2 n_{samples}} \|X w - y ||^2_2 + \alpha \|w\|_1 $$
and is popular for prediction as it simultaneously selects features thanks to the $\ell_1$-term. The greater $\alpha$ the fewer features.
Step15:  We can observe the regressor $w$ provided by the model, notice the sparsity.
Step16:  We can observe which coefficients are put to $0$ and which ones are positively/negatively correlated.
Step17:  Let us take a look at our predictions.
Step18:  Regularization path
Selecting a good parameter $\alpha$ is the role of the data scientist. For instance, a easy way to do is the following.
<ASSISTANT_TASK:>
Python Code:
import numpy as np
import matplotlib.pyplot as plt
from sklearn.datasets import make_blobs
%matplotlib inline
# we create 40 separable points in R^2 around 2 centers (random_state=6 is a seed so that the set is separable)
X, y = make_blobs(n_samples=40, n_features=2, centers=2 , random_state=6)
print(X[:5,:],y[:5]) # print the first 5 points and labels
plt.scatter(X[:, 0], X[:, 1], c=y,  cmap=plt.cm.Paired)
Explanation: <table>
<tr>
<td width=15%><img src="./img/UGA.png"></img></td>
<td><center><h1>Introduction to Python for Data Sciences</h1></center></td>
<td width=15%><a href="http://www.iutzeler.org" style="font-size: 16px; font-weight: bold">Franck Iutzeler</a> </td>
</tr>
</table>
<br/><br/>
<center><a style="font-size: 40pt; font-weight: bold">Chap. 4 - Scikit Learn </a></center> 
<br/><br/>
2- Supervised Learning
In the session, we will investigate some examples on how to deal with popular learning problems using standard algorithms. Many other problems and algorithms exist so this course is not at all exhaustive. 
Classification
End of explanation
from sklearn.svm import SVC # Support vector classifier i.e. Classifier by SVM
modelSVMLinear = SVC(kernel="linear")
modelSVMLinear.fit(X,y)
Explanation: Support Vector Machines (SVM) are based on learning a vector $w$ and an intercept $b$ such that the hyperplane $w^T x - b = 0$ separates the data i.e.  $a$ belongs to one class if  $w^T a - b > 0$ and the other elsewhere. 
They were later extended to Kernel methods that is  $\kappa(w, a) - b = 0$ is now the separating curve where $\kappa$ is the kernel, typically:
* linear: $\kappa(x,y)= x^T y$ (original SVM) 
* polynomial: $\kappa(x,y)= (x^T y)^d$ 
* Gaussian radial basis function (rfb):   $\kappa(x,y)= \exp( - \gamma \| x - y \|^2 )$
End of explanation
def plot_svc_decision_function(model, ax=None, plot_support=True):
    Plot the decision function for a 2D SVC
    if ax is None:
        ax = plt.gca()
    xlim = ax.get_xlim()
    ylim = ax.get_ylim()
    
    # create grid to evaluate model
    x = np.linspace(xlim[0], xlim[1], 30)
    y = np.linspace(ylim[0], ylim[1], 30)
    Y, X = np.meshgrid(y, x)
    xy = np.vstack([X.ravel(), Y.ravel()]).T
    P = model.decision_function(xy).reshape(X.shape)
    
    # plot decision boundary and margins
    ax.contour(X, Y, P, colors='k',
               levels=[-1, 0, 1], alpha=0.5,
               linestyles=['--', '-', '--'])
    
    # plot support vectors
    if plot_support:
        ax.scatter(model.support_vectors_[:, 0],
                   model.support_vectors_[:, 1],
                   s=300, linewidth=1, facecolors='none');
    ax.set_xlim(xlim)
    ax.set_ylim(ylim)
    
plt.scatter(X[:, 0], X[:, 1], c=y ,  cmap=plt.cm.Paired)
plot_svc_decision_function(modelSVMLinear)
Explanation: The following illustration can be found in the Python Data Science Handbook by Jake VanderPlas.
End of explanation
# we create points in R^2 around 2 centers (random_state=48443 is a seed so that the set is *not* separable)
X, y = make_blobs(n_samples=100, n_features=2, centers=2 , random_state=48443)
plt.scatter(X[:, 0], X[:, 1], c=y,  cmap=plt.cm.Paired)
Explanation: We see clearly that the linear SVM seeks at maximizing the margin between the hyperplane and the two well defined classes from the data.
Non-separable data
In real cases, the data is usually not linearly separable as before.
End of explanation
from sklearn.model_selection import train_test_split    # sklearn > ...
    
XTrain, XTest, yTrain, yTest = train_test_split(X,y,test_size = 0.5) # split data in two
model1 = SVC(kernel="linear",C=0.01)
model1.fit(XTrain,yTrain)
model2 = SVC(kernel="linear",C=100)
model2.fit(XTrain,yTrain)
plt.scatter(XTrain[:, 0], XTrain[:, 1], c=yTrain ,  cmap=plt.cm.Paired)
plot_svc_decision_function(model1)
plt.title("C = 0.01")
plt.scatter(XTrain[:, 0], XTrain[:, 1], c=yTrain ,  cmap=plt.cm.Paired)
plot_svc_decision_function(model2)
plt.title("C = 100")
Explanation: Let us use the same linear SVM classifier. Obviously, there are misclassified points, the model is thus learnt not by maximizing the margin (which does not exist anymore) but by minimizing a penalty over misclassified data. This penalty takes the form of an allowance margin controlled by a parameter $C$. The smaller $C$ the more inclusive the margin. Finding a good value for $C$ is up to the data scientist.
End of explanation
from sklearn.metrics import confusion_matrix
yFit1 = model1.predict(XTest)
yFit2 = model2.predict(XTest)
mat1 = confusion_matrix(yTest, yFit1)
mat2 = confusion_matrix(yTest, yFit2)
print('Model with C = 0.01')
print(mat1)
print("Model with C = 100")
print(mat2)
Explanation: To find out which value of $C$ to use or globally the performance of the classifier, one can use Scikit Learn's classification metrics, for instance the confusion matrix.
End of explanation
import seaborn as sns
sns.heatmap(mat1, square=True, annot=True ,cbar=False)
plt.ylabel('true label')
plt.xlabel('predicted label')
Explanation: It can also be plotted in a fancier way with seaborn.
End of explanation
from sklearn.datasets import make_moons
X,y = make_moons(noise=0.1)
plt.scatter(X[:, 0], X[:, 1], c=y,  cmap=plt.cm.Paired)
modelLinear = SVC(kernel="linear")
modelLinear.fit(X,y)
modelRbf = SVC(kernel="rbf")
modelRbf.fit(X,y)
plt.scatter(X[:, 0], X[:, 1], c=y,  cmap=plt.cm.Paired)
plot_svc_decision_function(modelLinear)
plot_svc_decision_function(modelRbf)
plt.title("The two models superposed")
Explanation: Kernels
When the separation between classes is not linear, kernels may be used to draw separating curves instead of lines. The most popular is the Gaussian rbf.
End of explanation
from sklearn.metrics import zero_one_loss
yFitLinear = modelLinear.predict(X)
yFitRbf = modelRbf.predict(X)
print("0/1 loss -- Linear: {:.3f}      Rbf: {:.3f}".format(zero_one_loss(y, yFitLinear),zero_one_loss(y, yFitRbf)))
Explanation: Let us compare the linear and rbf training error using the zero one loss (the proportion of misclassified examples).
End of explanation
import pandas as pd
import numpy as np
iris = pd.read_csv('data/iris.csv')
classes = pd.DataFrame(iris["species"])
features = iris.drop(["species","sepal_length","sepal_width"],axis=1)
classes.sample(6)
features.sample(6)
XTrain, XTest, yTrain, yTest = train_test_split(features,classes,test_size = 0.5)
from sklearn.multiclass import OneVsRestClassifier
yPred = OneVsRestClassifier(SVC()).fit(XTrain, yTrain).predict(XTest)
print(yPred)  # Note the classes are not number but everything went as expected
class_labels= ['virginica' , 'setosa' , 'versicolor']
sns.heatmap(confusion_matrix(yTest, yPred), square=True, annot=True ,cbar=False,  xticklabels= class_labels,  yticklabels=class_labels)
plt.ylabel('true label')
plt.xlabel('predicted label')
Explanation: Multiple classes
Where there are multiples classes (as in the iris dataset of the Pandas notebook), different strategies can be adopted:
* Transforming the multiclass problem into a binary one by looking at the one-vs-rest problem (for each class construct a binary classifier between it and the rest) or the one-vs-one one (where each couple of classes is considered separately). After this transformation, standard binary classifiers can be used.
* Using dedicated algorithms such as decision trees 
The corresponding algorithms can be found in the multiclass module documentation.
We are going to illustrate this by the iris 3-class classification problem using only the 2 petal features (width and length, this is only so that the feature vector is 2D and easy to visualize).
End of explanation
import pandas as pd
import numpy as np
student = pd.read_csv('data/student-mat.csv')
student.head()
target = pd.DataFrame(student["G3"])
features = student.drop(["G3"],axis=1)
Explanation: Other classifiers
The main classifiers from Scikit learn are:  Linear SVM, RBF SVM (as already seen), Nearest Neighbors, Gaussian Process, Decision Tree, Random Forest, Neural Net, AdaBoost, Naive Bayes, QDA.
Use is:
from sklearn.neural_network import MLPClassifier
from sklearn.neighbors import KNeighborsClassifier
from sklearn.svm import SVC
from sklearn.gaussian_process import GaussianProcessClassifier
from sklearn.gaussian_process.kernels import RBF
from sklearn.tree import DecisionTreeClassifier
from sklearn.ensemble import RandomForestClassifier, AdaBoostClassifier
from sklearn.naive_bayes import GaussianNB
from sklearn.discriminant_analysis import QuadraticDiscriminantAnalysis
classifiers = [
KNeighborsClassifier(3),
SVC(kernel="linear", C=0.025),
SVC(gamma=2, C=1),
GaussianProcessClassifier(1.0 * RBF(1.0), warm_start=True),
DecisionTreeClassifier(max_depth=5),
RandomForestClassifier(max_depth=5, n_estimators=10, max_features=1),
MLPClassifier(alpha=1),
AdaBoostClassifier(),
GaussianNB(),
QuadraticDiscriminantAnalysis()]
Regression
Let consider the problem of predicting real values from a set of features.
We will consider the <a href="http://archive.ics.uci.edu/ml/datasets/Student+Performance">student performance</a> dataset. The goal is to predict the final grade from the other information, we get from the documentation:
End of explanation
from sklearn.preprocessing import LabelEncoder
lenc = LabelEncoder()
num_features = features.apply(lenc.fit_transform)
num_features.head() 
Explanation: One immediate problem here is that the features are not numeric (not floats). Thankfully, Scikit Learn provides encoders to convert categorical (aka nominal, discrete) features to numerical ones.
End of explanation
from sklearn.preprocessing import StandardScaler, add_dummy_feature
scaler = StandardScaler()
normFeatures = add_dummy_feature(scaler.fit_transform(num_features))
preproData = pd.DataFrame(normFeatures , columns=[ "intercept" ] + list(num_features.columns) )
preproData.describe().T
Explanation: Even numerical values were encoded, as we are going to normalize, it is not really important. 
The normalization is done by removing the mean and equalizing the variance per feature, in addition, we are going to add an intercept.
End of explanation
from sklearn.model_selection import train_test_split    # sklearn > ...
    
from sklearn.linear_model import Lasso
XTrain, XTest, yTrain, yTest = train_test_split(preproData,target,test_size = 0.25)
model = Lasso(alpha=0.1)
model.fit(XTrain,yTrain)
Explanation: Regression and Feature selection with the Lasso
The lasso problem is finding a regressor $w$ such that minimizes
$$  \frac{1}{2 n_{samples}} \|X w - y ||^2_2 + \alpha \|w\|_1 $$
and is popular for prediction as it simultaneously selects features thanks to the $\ell_1$-term. The greater $\alpha$ the fewer features.
End of explanation
model.coef_
Explanation: We can observe the regressor $w$ provided by the model, notice the sparsity.
End of explanation
print("Value      Feature")
for idx,val in enumerate(model.coef_):
    print("{:6.3f}      {}".format(val,preproData.columns[idx]))
Explanation: We can observe which coefficients are put to $0$ and which ones are positively/negatively correlated.
End of explanation
targetPred = model.predict(XTest)
print("Predicted    True")
for idx,val in enumerate(targetPred):
    print("{:4.1f}          {:.0f}".format(val,float(yTest.iloc[idx])))
Explanation: Let us take a look at our predictions.
End of explanation
n_test = 15
alpha_tab = np.logspace(-10,1,base=2,num = n_test)
print(alpha_tab)
trainError = np.zeros(n_test)
testError = np.zeros(n_test)
featureNum = np.zeros(n_test)
for idx,alpha in enumerate(alpha_tab):
    model = Lasso(alpha=alpha)
    model.fit(XTrain,yTrain)
    yPredTrain = model.predict(XTrain)
    yPredTest = model.predict(XTest)
    
    trainError[idx] = np.linalg.norm(yPredTrain-yTrain["G3"].values)/yTrain.count()
    testError[idx] = np.linalg.norm(yPredTest-yTest["G3"].values)/yTest.count()
    featureNum[idx] = sum(model.coef_!=0)
    
alpha_opt = alpha_tab[np.argmin(testError)]
import matplotlib.pyplot as plt
import seaborn as sns
sns.set()
%matplotlib inline
plt.subplot(311)
plt.xscale("log")
plt.plot(alpha_tab, trainError,label="train error")
plt.xlim([min(alpha_tab),max(alpha_tab)])
plt.legend()
plt.xticks([])
plt.axvline(x=alpha_opt)
plt.ylabel("error")
plt.subplot(312)
plt.xscale("log")
plt.plot(alpha_tab, testError,'r',label="test error")
plt.xlim([min(alpha_tab),max(alpha_tab)])
#plt.ylim([0.19, 0.21])
plt.legend()
plt.axvline(x=alpha_opt)
plt.xticks([])
plt.ylabel("error")
plt.subplot(313)
plt.xscale("log")
plt.scatter(alpha_tab, featureNum)
plt.xlim([min(alpha_tab),max(alpha_tab)])
plt.ylim([0,28])
plt.axvline(x=alpha_opt)
plt.ylabel("nb. of features")
plt.xlabel("alpha")
Explanation: Regularization path
Selecting a good parameter $\alpha$ is the role of the data scientist. For instance, a easy way to do is the following.
End of explanation
<END_TASK>