<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 Support Vector Machines
implemented with sklearn (sci-kit learn), Python
Step1:  Following ex6.pdf of Programming Exercise 6
Step2:  Part 2
Step3:  You should try to change the $C$ value below and see how the decision boundary varies (e.g., try $C=1000$)
Step4:  cf. sklearn.svm.SVC
fit(X,y,sample_weight=None) 
X 
Step5:  Plotting
cf. Plot different SVM classifiers in the iris dataset
Step7:  Part 3
Step8:  1.2.2 Example Dataset 2, 1.2.3 Example Dataset 3
Step9:  Part 4
Step10:  Part 5
Step11:  Indeed
Step13:  Spam Classification (2. Spam Classification of ex6)
Part 1
Step14:  Dataset examples from sci-kit learn
cf. Non-linear SVM
Step15:  cf. SVM
<ASSISTANT_TASK:>
Python Code:
%matplotlib inline
import matplotlib.pyplot as plt
import sklearn
from sklearn import datasets
from sklearn import svm
from sklearn.feature_extraction.text import CountVectorizer
import nltk
import numpy as np
import scipy
import re
import os, sys
print(os.getcwd())
os.listdir( os.getcwd()+"/ex6/" ) 
# Import BeautifulSoup into your workspace
from bs4 import BeautifulSoup    
Explanation: Support Vector Machines
implemented with sklearn (sci-kit learn), Python
End of explanation
# Load saved matrices from file 
ex6data1_mat_data = scipy.io.loadmat( os.getcwd()+"/ex6/ex6data1.mat")
print(type(ex6data1_mat_data))
print(ex6data1_mat_data.keys())
print(type(ex6data1_mat_data["y"]))
print(ex6data1_mat_data['y'].shape)
print(type(ex6data1_mat_data["X"]))
print(ex6data1_mat_data['X'].shape)
plt.scatter( ex6data1_mat_data['X'][:,0] ,ex6data1_mat_data['X'][:,1] )
y=ex6data1_mat_data['y']
np.where(y==1)
# these are the x-coordinates of the X input data such that y=1
# ex6data1_mat_data['X'][np.where(y==1),0] 
# and so 
plt.scatter( ex6data1_mat_data['X'][np.where(y==0)[0],0], ex6data1_mat_data['X'][np.where(y==0)[0],1] ,
            s=35,c='y',marker='o' , label='y=0' )
plt.scatter( ex6data1_mat_data['X'][np.where(y==1)[0],0], ex6data1_mat_data['X'][np.where(y==1)[0],1] ,
            s=75,c='b',marker='+' , label='y=1' )
plt.legend(loc=6)
plt.show()
Explanation: Following ex6.pdf of Programming Exercise 6: Support Vector Machines of Coursera's Machine Learning Introduction by Andrew Ng
Part 1: Loading and Visualizing Data
We start the exercise by first loading and visualizing the dataset.
The following code will load the dataset into your environmenet and plot the data.
End of explanation
ex6data1_mat_data = scipy.io.loadmat( os.getcwd()+"/ex6/ex6data1.mat")
print(type(ex6data1_mat_data))
print(ex6data1_mat_data.keys())
print("\nTraining Linear SVM ...\n")
Explanation: Part 2: Training Linear SVM
The following code will train a linear SVM on the dataset and plot the decision boundary learned
Load from ex6data1
End of explanation
C=1
clf=svm.SVC()  # C=1. default, kernel='rbl' default, gamma : float, (default='auto') 
                # if gamma is 'auto' then 1/n_features will be used instead
Explanation: You should try to change the $C$ value below and see how the decision boundary varies (e.g., try $C=1000$)
End of explanation
print( ex6data1_mat_data['X'].shape )
print( ex6data1_mat_data['y'].shape )
clf.fit( ex6data1_mat_data['X'], ex6data1_mat_data['y'].flatten())
# get support vectors
clf.support_vectors_
# get indices of support vectors
clf.support_
# get number of support vectors for each class
clf.n_support_
Explanation: cf. sklearn.svm.SVC
fit(X,y,sample_weight=None) 
X : (array-like, sparse matrix), shape(n_samples,n_features), 
y : array-like,shape(n_samples)
End of explanation
h=.02 # step size in the mesh
# create a mesh to plot in
X = ex6data1_mat_data['X']
x_min, x_max = X[:,0].min()-1, X[:,0].max()+1
y_min, y_max = X[:,1].min()-1, X[:,1].max()+1
xx,yy = np.meshgrid(np.arange(x_min,x_max,h), np.arange(y_min,y_max,h))
Z = clf.predict(np.c_[xx.ravel(),yy.ravel()])  # translates slice objects to concatenation along the second axis
print(Z.shape)
Z = Z.reshape(xx.shape)
plt.contourf(xx,yy,Z, cmap=plt.cm.coolwarm, alpha=0.8)
# Plot also the training points
plt.scatter(X[:,0],X[:,1],c=ex6data1_mat_data['y'], cmap=plt.cm.coolwarm)
print(xx.shape); print(Z.shape)
clf_lin=svm.SVC(kernel='linear',C=C).fit(ex6data1_mat_data['X'], ex6data1_mat_data['y'].flatten())
h=.02 # step size in the mesh
# create a mesh to plot in
X = ex6data1_mat_data['X']
x_min, x_max = X[:,0].min()-1, X[:,0].max()+1
y_min, y_max = X[:,1].min()-1, X[:,1].max()+1
xx,yy = np.meshgrid(np.arange(x_min,x_max,h), np.arange(y_min,y_max,h))
Z = clf_lin.predict(np.c_[xx.ravel(),yy.ravel()])  # translates slice objects to concatenation along the second axis
print(Z.shape)
Z = Z.reshape(xx.shape)
plt.contourf(xx,yy,Z, cmap=plt.cm.coolwarm, alpha=0.8)
# Plot also the training points
plt.scatter(X[:,0],X[:,1],c=ex6data1_mat_data['y'], cmap=plt.cm.coolwarm)
C_lst = [0.01,0.1,1.,100.]
clf_lst = [svm.SVC(kernel='linear',C=C).fit(ex6data1_mat_data['X'], ex6data1_mat_data['y'].flatten()) for C in C_lst]
h=.02 # step size in the mesh
# create a mesh to plot in
X = ex6data1_mat_data['X']
x_min, x_max = X[:,0].min()-1, X[:,0].max()+1
y_min, y_max = X[:,1].min()-1, X[:,1].max()+1
xx,yy = np.meshgrid(np.arange(x_min,x_max,h), np.arange(y_min,y_max,h))
# title for the plots
titles = ['SVC linear kernel C='+str(C) for C in C_lst]
for i, clf in enumerate(clf_lst):
    # Plot the decision boundary.  For that, we'll assign a color to each 
    # point in the mesh [x_min, x_max]x[y_min, y_max]
    plt.subplot(2,2,i+1)
    
    Z=clf.predict(np.c_[xx.ravel(), yy.ravel()])
    
    # Put the result into a color plot 
    Z = Z.reshape(xx.shape)
    plt.contourf(xx,yy,Z,cmap=plt.cm.coolwarm, alpha=0.8)
    
    # Plot also the training points
    plt.scatter(X[:,0], X[:,1],c=ex6data1_mat_data['y'], cmap=plt.cm.coolwarm)
    plt.title(titles[i])
plt.show()
    
titles
Explanation: Plotting
cf. Plot different SVM classifiers in the iris dataset
End of explanation
x1=np.array([1,2,1])
x2=np.array([0,4,-1])
sigma=2
sum( -(x1-x2)**2 )/(2.*2**2)
np.exp( sum( -(x1-x2)**2 )/(2.*2**2) ) # 
def gaussianKernel(x1,x2,sigma):
     gaussianKernel : returns a gaussian kernel between x1 and x2 and returns the value in sim 
    
    
    # You need to return the following variables correctly.
    sim = 0
    
    sim = np.exp( -np.sum((x1-x2)**2/(2.*sigma**2)))
    return sim
    
gaussianKernel(x1,x2,sigma)
Explanation: Part 3: Implementing Gaussian Kernel
You will now implement the Gaussian kernel to use with the SVM.  You should complete the code in gaussianKernel.m
End of explanation
# Load saved matrices from file 
ex6data2_mat_data = scipy.io.loadmat( os.getcwd()+"/ex6/ex6data2.mat")
print(type(ex6data2_mat_data))
print(ex6data2_mat_data.keys())
ex6data3_mat_data = scipy.io.loadmat( os.getcwd()+"/ex6/ex6data3.mat")
print(type(ex6data3_mat_data))
print(ex6data3_mat_data.keys())  # Xval, yval are CROSS VALIDATION set data
Explanation: 1.2.2 Example Dataset 2, 1.2.3 Example Dataset 3
End of explanation
X = ex6data2_mat_data['X']
print(X.shape)
plt.scatter(X[:,0],X[:,1],c=ex6data2_mat_data['y'], cmap=plt.cm.coolwarm)
Explanation: Part 4: Visualizing Dataset 2
The following code will load the next dataset into your environment and plot the data.
End of explanation
C=1.
sigma=0.1
gamma_gaussiankernel = 1./(2.*sigma**2)  # I'm supposing that sci-kit learn SVC's gamma = 1/(2*sigma^2)
clf=svm.SVC(kernel='rbf',C=C,gamma=gamma_gaussiankernel)
clf.fit( ex6data2_mat_data['X'], ex6data2_mat_data['y'].flatten())
h=.02 # step size in the mesh
# create a mesh to plot in
X = ex6data2_mat_data['X']
x_min, x_max = X[:,0].min()-.1, X[:,0].max()+.1
y_min, y_max = X[:,1].min()-.1, X[:,1].max()+.1
xx,yy = np.meshgrid(np.arange(x_min,x_max,h), np.arange(y_min,y_max,h))
Z = clf.predict(np.c_[xx.ravel(),yy.ravel()])  # translates slice objects to concatenation along the second axis
print(Z.shape)
Z = Z.reshape(xx.shape)
plt.contourf(xx,yy,Z, cmap=plt.cm.coolwarm, alpha=0.8)
# Plot also the training points
plt.scatter(X[:,0],X[:,1],c=ex6data2_mat_data['y'], cmap=plt.cm.coolwarm)
X = ex6data3_mat_data['X']
y = ex6data3_mat_data['y']
Xval = ex6data3_mat_data['Xval']
yval = ex6data3_mat_data['yval']
C_lst = [0.0001,0.001,0.003,0.01,0.03,0.1,0.3,1.,10.]
sigma_lst = [0.0001,0.001,0.003,0.01,0.03,0.1,0.3,1.,10.]
models = [ [svm.SVC(kernel='rbf',
                   C=C,
                   gamma=1./(2.*sigma**2)).fit(X,
                                               y.flatten()) for C in C_lst] for sigma in sigma_lst]
ex6data3_mat_data.keys()
(models[0][0].predict(Xval) != yval).astype('int').mean()
predict_errs = np.array( [[(model.predict(Xval)!=yval).astype('int').mean() for model in rowmodel] for rowmodel in models] )
predict_errs
predict_errs[predict_errs.argmin() // 9 , predict_errs.argmin() % 9]
print( sigma_lst[predict_errs.argmin()//9] )
print( C_lst[predict_errs.argmin() % 9] )
Explanation: Part 5: Training SVM with RBF Kernel (Dataset 2)
After you have implemented the kernel, we can now use it to train the SVM classifier.
End of explanation
C=1.0
sigma=0.03
clf = svm.SVC(kernel='rbf',C=C,gamma=1./(2.*sigma**2)).fit(X,y.flatten())
(clf.predict(Xval) != yval).astype('int').mean()
Explanation: Indeed
End of explanation
# Extract Features
f = open(os.getcwd()+"/ex6/emailSample1.txt",'r')
file_contents = f.read()
f.close()
file_contents
# Lower case
file_contents.lower()
# Strip all HTML
# Looks for any expression that starts with < and ends with > and replace 
# and does not have any < or > in the tag it with a space
BeautifulSoup( file_contents.lower() )
BeautifulSoup( file_contents.lower() ).get_text()
# Calling get_text() gives you the text of the review, without tags or markup.
# cf. https://www.kaggle.com/c/word2vec-nlp-tutorial/details/part-1-for-beginners-bag-of-words
import re
# Use regular expressions to do a find-and-replace
# Handle Numbers
# look for 1 or more characters between 0-9
email_contents = re.sub("[0-9]+",           # The pattern to search for
                      "number",                   # The pattern to replace it with
                      BeautifulSoup( file_contents.lower() ).get_text() )  # The text to search
# Handle URLS
# Look for strings starting with http:// or https://
re.sub( '(http|https)://[^\s]*','httpaddr',email_contents)
# Handle Email Addresses
# Look for strings with @ in the middle
re.sub( '[^\s]+@[^\s]+','emailaddr', email_contents)
# Handle $ sign
re.sub('[$]+','dollar', email_contents)
def processEmail_regex(email_contents):
     processEmail_regex - process email with regular expressions, 1st.
    
    # Lower case
    email_contents = email_contents.lower()
    
    # Strip all HTML
    # Looks for any expression that starts with < and ends with > and replace 
    # and does not have any < or > in the tag it with a space
    email_contents = BeautifulSoup( email_contents,"lxml" ).get_text()
    
    # Use regular expressions to do a find-and-replace
    # Handle Numbers
    # look for 1 or more characters between 0-9
    email_contents = re.sub("[0-9]+",           # The pattern to search for
                      "number",                   # The pattern to replace it with
                      email_contents )  # The text to search
    
    # Handle URLS
    # Look for strings starting with http:// or https://
    email_contents = re.sub( '(http|https)://[^\s]*','httpaddr',email_contents)
    
    # Handle Email Addresses
    # Look for strings with @ in the middle
    email_contents = re.sub( '[^\s]+@[^\s]+','emailaddr', email_contents)
    
    # Handle $ sign
    email_contents = re.sub('[$]+','dollar', email_contents)
    
    # Remove any non alphanumeric characters
    email_contents = re.sub('[^a-zA-Z0-9]',' ', email_contents)
    
    return email_contents
    
f = open(os.getcwd()+"/ex6/emailSample1.txt",'r')
file_contents = f.read()
f.close()
email_contents = processEmail_regex(file_contents)
email_contents
from sklearn.feature_extraction.text import CountVectorizer
count_vect = CountVectorizer()
test_email_vec = count_vect.fit_transform( [email_contents,])
print( type(test_email_vec) ); 
print( test_email_vec.shape )
test_email_vec[0][0]
import nltk
nltk.download()
tokens_email_contents = nltk.word_tokenize(email_contents)
email_contents
tokens_email_contents
tagged_email_contents = nltk.pos_tag( tokens_email_contents )
tagged_email_contents
entities_email_contents = nltk.chunk.ne_chunk( tagged_email_contents )
type(entities_email_contents)
entities_email_contents
Explanation: Spam Classification (2. Spam Classification of ex6)
Part 1: Email Preprocessing
To use an SVM to classify emails into Spam vs. Non-Spam, you first need to convert each email into a vector of features.  In this part, you will implement the preprocessing steps for each email.
End of explanation
X = np.random.randn(300,2)
y = np.logical_xor(X[:,0] > 0, X[:,1] > 0)
print(X.shape)
print(y.shape)
print(X.max())
print(X.min())
print(y.max()); print(y.min())
plt.scatter(X[:,0],X[:,1],s=30,c=y,cmap=plt.cm.Paired)
y = y.astype("int")
print(y.max());print(y.min())
plt.scatter(X[:,0],X[:,1],s=30,c=y,cmap=plt.cm.Paired)
Explanation: Dataset examples from sci-kit learn
cf. Non-linear SVM
End of explanation
# we create 40 separable points
np.random.seed(0)
X = np.r_[np.random.randn(20, 2) - [2, 2], np.random.randn(20, 2) + [2, 2]]
y = [0] * 20 + [1] * 20 
plt.scatter(X[:, 0], X[:, 1], c=y, cmap=plt.cm.Paired)
Explanation: cf. SVM: Maximum margin separating hyperplane
End of explanation
<END_TASK>