<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 Exercise 6 | Support Vector Machines
Step1:  Part 1
Step2:  Part 2
Step3:  Part 3
Step4:  The Gaussian Kernel between x1 = [1; 2; 1], x2 = [0; 4; -1], sigma = 2 should be about 0.324652.
Step5:  Part 4
Step6:  Part 5
Step7:  Part 6
Step8:  Part 7
<ASSISTANT_TASK:>
Python Code:
import numpy as np
import matplotlib.pyplot as plt
import scipy.io
import sklearn.svm
%matplotlib inline
Explanation: Exercise 6 | Support Vector Machines
End of explanation
ex6data1 = scipy.io.loadmat('ex6data1.mat')
X = ex6data1['X']
y = ex6data1['y'][:, 0]
def plot_data(X, y, ax=None):
    if ax == None:
        fig, ax = plt.subplots(figsize=(7,5))
    pos = y==1
    neg = y==0
    ax.scatter(X[pos,0], X[pos,1], marker='+', color='b')
    ax.scatter(X[neg,0], X[neg,1], marker='o', color='r', s=5)
plot_data(X, y)
Explanation: Part 1: Loading and Visualizing Data
We start the exercise by first loading and visualizing the dataset. 
  The following code will load the dataset into your environment and plot
  the data.
End of explanation
svm = sklearn.svm.SVC(C=1, kernel='linear')
svm.fit(X, y)
np.mean(svm.predict(X) == y)
svm.coef_
fig, ax = plt.subplots(figsize=(7,5))
def draw_contour(X, model):
    x1 = np.linspace(np.min(X[:,0]), np.max(X[:,0]), 200)
    x2 = np.linspace(np.min(X[:,1]), np.max(X[:,1]), 200)
    xx1, xx2 = np.meshgrid(x1, x2)
    yy = model.predict(np.c_[xx1.flat, xx2.flat]).reshape(xx1.shape)
    ax.contour(x1, x2, yy, levels=[0.5])
    
plot_data(X, y, ax)
draw_contour(X, svm)
Explanation: Part 2: Training Linear SVM
The following code will train a linear SVM on the dataset and plot the
  decision boundary learned.
You should try to change the C value below and see how the decision
 boundary varies (e.g., try C = 1000)
End of explanation
def gaussianKernel(x1, x2, sigma):
    # ====================== YOUR CODE HERE ======================
    # Instructions: Fill in this function to return the similarity between x1
    #               and x2 computed using a Gaussian kernel with bandwidth
    #               sigma
    #
    #
    
    return 0
    
    # =============================================================
    
Explanation: Part 3: Implementing Gaussian Kernel
You will now implement the Gaussian kernel to use
  with the SVM. You should complete the code in gaussianKernel. This notebook will not use it, however. An sklearn custom kernel should return a matrix of all kernel values. Feel free to implement gaussianKernel in the sklearn way, and later call svm.SVC(kernel=gaussianKernel).
End of explanation
gaussianKernel(x1=np.array([1, 2,  1]), x2=np.array([0, 4, -1]), sigma=2)
Explanation: The Gaussian Kernel between x1 = [1; 2; 1], x2 = [0; 4; -1], sigma = 2 should be about 0.324652.
End of explanation
ex6data2 = scipy.io.loadmat('ex6data2.mat')
X = ex6data2['X']
y = ex6data2['y'][:,0]
print(X.shape, y.shape)
plot_data(X, y)
Explanation: Part 4: Visualizing Dataset 2
The following code will load the next dataset into your environment and 
  plot the data.
End of explanation
model = sklearn.svm.SVC(C=1, gamma=100, kernel='rbf')
model.fit(X, y)
np.mean((model.predict(X) == y))
fig, ax = plt.subplots()
plot_data(X, y, ax)
draw_contour(X, model)
Explanation: Part 5: Training SVM with RBF Kernel (Dataset 2)
After you have implemented the kernel, we can now use it to train the 
  SVM classier.
Note that this doesn't do this, it simply uses the built-in gaussian kernel in sklearn.
End of explanation
ex6data3 = scipy.io.loadmat('ex6data3.mat')
X = ex6data3['X']
y = ex6data3['y'][:, 0]
Xval = ex6data3['Xval']
yval = ex6data3['yval'][:, 0]
print(X.shape, y.shape, Xval.shape, yval.shape)
plot_data(X, y)
plot_data(Xval, yval)
Explanation: Part 6: Visualizing Dataset 3
The following code will load the next dataset into your environment and 
  plot the data.
End of explanation
import itertools
possible_C = [0.01, 0.03, 0.1, 0.3, 1, 3, 10, 30, 100, 300, 1000]
possible_gamma = [0.01, 0.03, 0.1, 0.3, 1, 3, 10, 30, 100, 300, 1000]
cv_errors = np.zeros((len(possible_C), len(possible_gamma)))
# YOUR CODE GOES HERE
C = 7
gamma = 7
# ==================
model = sklearn.svm.SVC(C=C, gamma=gamma, kernel='rbf')
model.fit(X, y)
fig, ax = plt.subplots()
plot_data(X, y, ax)
draw_contour(X, model)
Explanation: Part 7: Training SVM with RBF Kernel (Dataset 3)
This is a different dataset that you can use to experiment with. Try
  different values of C and sigma here, train a classifier on your training data, measure the cross validation error and find the values for C and sigma that minimize the cross validation error.
End of explanation
<END_TASK>