<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 Linear Support Vector Classifier
Support vector machines are a set of supervised learning algorithms that you can use for classification, regression and outlier detection purposes. SciKit-Learn has many classes for SVM usage, depending on your purpose. The one we'll be focusing on is Support Vector Classifier, SVC.
An OCR example
In 1982, the first computer-driven, OCR machine got installed by  the United States Postal Service (USPS) in Los Angeles and by the end of 1984, over 250 OCRs machines were installed in 118 major  mail processing centers across the country.
Let's see if it's possible to train a support vector classifier in a few seconds using machine learning, and if the classification accuracy is similar or better  than the advertised USPS stats.   
We start by reading the dataset, which comes from the UCI Machine Learning Repository and is composed by bitmaps of handwritten digits from a preprinted form.
Step1:  Let's have a look at these bitmaps of handwritten digits
Step2:  Train the SVM Classifier
Now we are ready to train the Support Vector Classifier, using the SciKitLearn library.
We leave all parameters at their defaults, setting only the kernel to be Linear.
More on the kernels later in this notebook.
Step3:  Checkpoint
Print the predicted digit and the actual label for a random example. 
We take the thousandth digit.
Step4:  The model's prediction was correct.
We can display that image, so we can visually check if it was a hard image or an easy image
Step5:  visual confirmation of accuracy
Here we can print more digits with indication of what was the predicted label (in red if it was wrong)
Step6:  Score
We can see that - on this sample of 50 handwritten digits - 4 of them are wrong, that's 8%.
And here we calculate the score on the entire testing dataset
Step7:  Not bad, the model was correct more than 96% of the times!
Non-linear Kernels for the SVC
We experiment now with different kernels, starting with the polynomial kernel. 
When training an SVM with a kernel, two additional parameters must be considered
Step8:  Which is sightly better, but we can try a different kernel, more performant
Step9:  Now it's better than the UPS' score!
Hyper-parameters tuning for SVC Kernels
Proper choice of C and gamma is critical to the SVM’s performance.
One is advised to use sklearn.model_selection.GridSearchCV with C and gamma spaced exponentially far apart to choose good values.
We will tune them - and the pre-processor - using a different example.
A Parkinson's classifier
Apply SVC to the Parkinson's Data Set, provided courtesy of UCI's Machine Learning Repository. The dataset was created at the University of Oxford, in collaboration with 10 medical centers around the US, along with Intel who developed the device used to record the primary features of the dataset
Step10:  We can apply different scaler for the pre-processing.
The standard scaler seems the best, feel free to experiment with others, uncommenting one below
Step11:  Same for the dimensionality reduction
Step12:  Train the SVM classifier.
Create and fit an SVC based on the RBF kernel against the training data and then finally score the testing data.  
To search the best hyper-parameters we just use simple nested loops.
Looping C from 0.05 to 2 and looping gamma from 0.001 to 0.1
Step13:  Best score was with C=0.85 and gamma=0.088
Comparing KNN vs. SVC
How does SVC compare with other classifiers, such as the KNN?
We classify the UCI's wheat-seeds dataset - that we used previously with the KNN algorithm - by using the SVC and compare the results.
First, benchmark how long it takes to train and predict with SVC relative to how long K-Neighbors took to train and test and then compare the decision boundary plot produced by the two.
Defining some parameters
Step14:  Read the data
Step15:  Data pre-processing
This is all standard.  You can refer to the previous examples for more details, especially the KNN example.
Step16:  Split into training and testing data sets
Step17:  Utility function
Step18:  Utility function
Step19:  Train the Knn classifier
Step20:  And get its benchmark
Step21:  Train the SVM Classifier
<ASSISTANT_TASK:>
Python Code:
import pandas as pd
# The Dataset comes from:
# https://archive.ics.uci.edu/ml/datasets/Optical+Recognition+of+Handwritten+Digits
  # Load up the data.
with open('../Datasets/optdigits.tes', 'r') as f: testing  = pd.read_csv(f)
with open('../Datasets/optdigits.tra', 'r') as f: training = pd.read_csv(f)
  # The number of samples between training and testing can vary
  # But the number of features better remain the same!
n_features = testing.shape[1]
X_test  = testing.iloc[:,:n_features-1]
X_train = training.iloc[:,:n_features-1]
y_test  = testing.iloc[:,n_features-1:].values.ravel()
y_train = training.iloc[:,n_features-1:].values.ravel()
print (n_features)
Explanation: Linear Support Vector Classifier
Support vector machines are a set of supervised learning algorithms that you can use for classification, regression and outlier detection purposes. SciKit-Learn has many classes for SVM usage, depending on your purpose. The one we'll be focusing on is Support Vector Classifier, SVC.
An OCR example
In 1982, the first computer-driven, OCR machine got installed by  the United States Postal Service (USPS) in Los Angeles and by the end of 1984, over 250 OCRs machines were installed in 118 major  mail processing centers across the country.
Let's see if it's possible to train a support vector classifier in a few seconds using machine learning, and if the classification accuracy is similar or better  than the advertised USPS stats.   
We start by reading the dataset, which comes from the UCI Machine Learning Repository and is composed by bitmaps of handwritten digits from a preprinted form.
End of explanation
import matplotlib.pyplot as plt
# The 'targets' or labels are stored in y. The 'samples' or data is stored in X
print ("Peeking the data...")
fig = plt.figure()
cnt = 0
for col in range(5):
    for row in range(10):
        plt.subplot(5, 10, cnt + 1)
        plt.imshow(X_train.iloc[cnt,:].values.reshape(8,8), cmap=plt.cm.gray_r, interpolation='nearest')
        plt.axis('off')
        cnt += 1
    fig.set_tight_layout(True)
plt.show()
Explanation: Let's have a look at these bitmaps of handwritten digits
End of explanation
from sklearn import svm # Library for Support Vector Machines
#
# Create and train an SVM classifier. 
print ("Training SV Classifier...")
svc = svm.SVC(kernel='linear')
svc.fit(X_train, y_train) 
Explanation: Train the SVM Classifier
Now we are ready to train the Support Vector Classifier, using the SciKitLearn library.
We leave all parameters at their defaults, setting only the kernel to be Linear.
More on the kernels later in this notebook.
End of explanation
#
# Print out the TRUE value of the 1000th digit in the test set
# By TRUE value, we mean, the actual provided label for that sample
#
true_1000th_test_value = y_test[999]
print ("1000th test label is: ", true_1000th_test_value)
#
# Predict the value of the 1000th digit in the test set.
# Was the model's prediction correct?
#
guess_1000th_test_value = svc.predict(X_test[999:1000])
print ("1000th test prediction is: ", guess_1000th_test_value)
Explanation: Checkpoint
Print the predicted digit and the actual label for a random example. 
We take the thousandth digit.
End of explanation
#
# Use IMSHOW to display the 1000th test image
# 
#
plt.imshow(X_test.iloc[999,:].values.reshape(8,8), cmap=plt.cm.gray_r, interpolation='nearest');
Explanation: The model's prediction was correct.
We can display that image, so we can visually check if it was a hard image or an easy image:
End of explanation
# Visual Confirmation of accuracy
fig = plt.figure()
  # Make some guesses
y_guess = svc.predict(X_test)
num_rows = 10
num_cols = 5
index = 0
for col in range(num_cols):
    for row in range(num_rows):
        plt.subplot(num_cols, num_rows, index + 1)
      # 8x8 is the size of the image, 64 pixels
        plt.imshow(X_test.iloc[index,:].values.reshape(8,8), cmap=plt.cm.gray_r, interpolation='nearest')
      # Green = Guessed right
      # Red = Fail!
        fontcolor = 'g' if y_test[index] == y_guess[index] else 'r'
        plt.title('Label: %i' % y_guess[index], fontsize=7, color=fontcolor)
        plt.axis('off')
        index += 1
fig.set_tight_layout(True)
plt.show()
Explanation: visual confirmation of accuracy
Here we can print more digits with indication of what was the predicted label (in red if it was wrong):
End of explanation
# Calculate the score of the SVC against the testing data
print ("Scoring SVM Classifier...")
#
score = svc.score(X_test, y_test)
print ("Score: ", score)
Explanation: Score
We can see that - on this sample of 50 handwritten digits - 4 of them are wrong, that's 8%.
And here we calculate the score on the entire testing dataset:
End of explanation
#
# We start with the POLY kernel
svc = svm.SVC(kernel='poly', C=1.0, gamma=0.001)
svc.fit(X_train, y_train) 
# Calculate the score of the SVC against the testing data
print ("Scoring SV poly Classifier...")
score = svc.score(X_test, y_test)
print ("Score: ", score)
Explanation: Not bad, the model was correct more than 96% of the times!
Non-linear Kernels for the SVC
We experiment now with different kernels, starting with the polynomial kernel. 
When training an SVM with a kernel, two additional parameters must be considered: C and gamma.
The parameter C, common to all SVM kernels, trades off misclassification of training examples against simplicity of the decision surface.
A low C makes the decision surface smooth, while a high C aims at classifying all training examples correctly.
We keep C at the default value = 1.  
Gamma defines how much influence a single training example has.
The larger gamma is, the closer other examples must be to be affected.
USPS has an advertised accuracy score of 98% which is higher than our SVC with a linear Kernel.
We can beat it with a non-linear kernel!
End of explanation
#
# change  SVC's kernel to 'rbf' 
svc = svm.SVC(kernel='rbf', C=1.0, gamma=0.001)
svc.fit(X_train, y_train) 
# Calculate the score of SVC against the testing data
print ("Scoring SVM rbf Classifier...")
score = svc.score(X_test, y_test)
print ("Score: ", score)
Explanation: Which is sightly better, but we can try a different kernel, more performant: the Radial Basis Function (RBF) kernel.
End of explanation
X = pd.read_csv("../Datasets/parkinsons.data")
X.drop(['name'], axis=1, inplace=True) # drop name column
y = X.status.copy() # copy “y” values out from status
X.drop(['status'], axis=1, inplace=True) # drop status column
# Perform a train/test split. 30% test group size, with a random_state equal to 7.
from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, 
                                                    random_state=7)
Explanation: Now it's better than the UPS' score!
Hyper-parameters tuning for SVC Kernels
Proper choice of C and gamma is critical to the SVM’s performance.
One is advised to use sklearn.model_selection.GridSearchCV with C and gamma spaced exponentially far apart to choose good values.
We will tune them - and the pre-processor - using a different example.
A Parkinson's classifier
Apply SVC to the Parkinson's Data Set, provided courtesy of UCI's Machine Learning Repository. The dataset was created at the University of Oxford, in collaboration with 10 medical centers around the US, along with Intel who developed the device used to record the primary features of the dataset: speech signals. 
Goals: first to see if it's possible to differentiate between people who have Parkinson's and who don't using SciKit-Learn's support vector classifier and then to take a first-stab at a naive way of fine-tuning our parameters in an attempt to maximize the accuracy of the testing set.  
Read and pre-process the data
End of explanation
from sklearn import preprocessing
  # tried with different scaler, standard is the best
    
scaler = preprocessing.StandardScaler() # best score was 0.932203389831
#scaler = preprocessing.MinMaxScaler()  # best score was 0.881355932203 
#scaler = preprocessing.MaxAbsScaler()  # best score was 0.881355932203 
#scaler = preprocessing.Normalizer()    # best score was 0.796610169492 
#scaler = preprocessing.KernelCenterer() # best score was 0.915254237288 
X_train = scaler.fit_transform(X_train)
X_test = scaler.transform(X_test)
Explanation: We can apply different scaler for the pre-processing.
The standard scaler seems the best, feel free to experiment with others, uncommenting one below:
End of explanation
from sklearn.decomposition import PCA
from sklearn import manifold
usePCA = False # change this to use PCA as dimensionality reducer
if usePCA:
    reducer = PCA(n_components=7).fit(X_train)
else:
    reducer = manifold.Isomap(n_neighbors=3, n_components=6).fit(X_train)
X_train = reducer.transform(X_train)
X_test  = reducer.transform(X_test)
Explanation: Same for the dimensionality reduction: feel free to experiment with PCA or Isomap
End of explanation
import numpy as np
  # a naive, best-parameter search using nested for-loops.
best_score = 0
for c in np.arange(0.05,2,0.05):
    for gamma in np.arange(0.001, 0.1, 0.001):
        svc = svm.SVC(kernel='rbf', C=c, gamma=gamma)
        svc.fit(X_train, y_train) 
        score = svc.score(X_test, y_test)
        if score > best_score:
            best_score = score
            #print ("New best score:", score, "using C= ", c, "and gamma = ", gamma)
            print(f"New best score: {score:.3f} using C=  {c:.2f} and gamma = {gamma:.3f}")
Explanation: Train the SVM classifier.
Create and fit an SVC based on the RBF kernel against the training data and then finally score the testing data.  
To search the best hyper-parameters we just use simple nested loops.
Looping C from 0.05 to 2 and looping gamma from 0.001 to 0.1
End of explanation
# 
# INFO:  Parameters can be adjusted here
C = 1
kernel = 'linear'
iterations = 100   
#
# INFO: You can set this to false if you want to
# draw the full square matrix
FAST_DRAW = True
Explanation: Best score was with C=0.85 and gamma=0.088
Comparing KNN vs. SVC
How does SVC compare with other classifiers, such as the KNN?
We classify the UCI's wheat-seeds dataset - that we used previously with the KNN algorithm - by using the SVC and compare the results.
First, benchmark how long it takes to train and predict with SVC relative to how long K-Neighbors took to train and test and then compare the decision boundary plot produced by the two.
Defining some parameters
End of explanation
# 
# Load up the wheat dataset into dataframe 'X'
#
df = pd.read_csv("../Datasets/wheat.data", index_col='id')
Explanation: Read the data
End of explanation
# INFO: An easy way to show which rows have nans in them
print (df[pd.isnull(df).any(axis=1)])
# 
# Go ahead and drop any row with a nan
#
df.dropna(axis=0, inplace=True)
# 
# INFO: you might try setting the nan values to the
# mean value of that column, the mean should only be calculated for
# the specific class rather than across all classes, now that you
# have the labels
#
# Copy the labels out of the dset into variable 'y' then Remove
# them from X. Encode the labels -- canadian:0, kama:1, and rosa:2
#
labels = df.wheat_type.copy() # copy “y” values out
df.drop(['wheat_type'], axis=1, inplace=True) # drop output column 
labels = labels.map({'canadian':0, 'kama':1, 'rosa':2})
Explanation: Data pre-processing
This is all standard.  You can refer to the previous examples for more details, especially the KNN example.
End of explanation
# 
# Split data into test / train sets
#
X_train, X_test, y_train, y_test = train_test_split(df, labels, test_size=0.3, 
                                                    random_state=7)
Explanation: Split into training and testing data sets
End of explanation
import matplotlib as mpl
import matplotlib.pyplot as plt
def drawPlots(model, X_train, X_test, y_train, y_test, wintitle='Figure 1'):
  # If this line throws an error, use plt.style.use('ggplot') instead
  mpl.style.use('ggplot') # Look Pretty
  padding = 3
  resolution = 0.5
  max_2d_score = 0
  score = 0
  y_colors = ['#ff0000', '#00ff00', '#0000ff']
  my_cmap = mpl.colors.ListedColormap(['#ffaaaa', '#aaffaa', '#aaaaff'])
  colors = [y_colors[i] for i in y_train]
  num_columns = len(X_train.columns)
  fig = plt.figure()
  fig.canvas.set_window_title(wintitle)
  
  cnt = 0
  for col in range(num_columns):
    for row in range(num_columns):
      # Easy out
      if FAST_DRAW and col > row:
        cnt += 1
        continue
      ax = plt.subplot(num_columns, num_columns, cnt + 1)
      plt.xticks(())
      plt.yticks(())
      # Intersection:
      if col == row:
        plt.text(0.5, 0.5, X_train.columns[row], verticalalignment='center', horizontalalignment='center', fontsize=12)
        cnt += 1
        continue
      # Only select two features to display, then train the model
      X_train_bag = X_train.iloc[:, [row,col]]
      X_test_bag = X_test.iloc[:, [row,col]]
      model.fit(X_train_bag, y_train)
      # Create a mesh to plot in
      x_min, x_max = X_train_bag.iloc[:, 0].min() - padding, X_train_bag.iloc[:, 0].max() + padding
      y_min, y_max = X_train_bag.iloc[:, 1].min() - padding, X_train_bag.iloc[:, 1].max() + padding
      xx, yy = np.meshgrid(np.arange(x_min, x_max, resolution),
                           np.arange(y_min, y_max, resolution))
      # Plot Boundaries
      plt.xlim(xx.min(), xx.max())
      plt.ylim(yy.min(), yy.max())
      # Prepare the contour
      Z = model.predict(np.c_[xx.ravel(), yy.ravel()])
      Z = Z.reshape(xx.shape)
      plt.contourf(xx, yy, Z, cmap=my_cmap, alpha=0.8)
      plt.scatter(X_train_bag.iloc[:, 0], X_train_bag.iloc[:, 1], c=colors, alpha=0.5)
      score = round(model.score(X_test_bag, y_test) * 100, 3)
      #plt.text(0.5, 0, "Score: {0}".format(score), transform = ax.transAxes, horizontalalignment='center', fontsize=8)
      plt.text(0.5, 0, f"Score: {score}", transform = ax.transAxes, horizontalalignment='center', fontsize=8)
      max_2d_score = score if score > max_2d_score else max_2d_score
      cnt += 1
  print ("Max 2D Score: ", max_2d_score)
  fig.set_tight_layout(True)
Explanation: Utility function: draw the plots
This is a convenience function to break any higher-dimensional space down and view cross sections of it.
End of explanation
import time
def benchmark(model, X_train, X_test, y_train, y_test, wintitle='Figure 1'):
  print ('\n\n' + wintitle + ' Results')
  
  # the only purpose of doing many iterations was to get a more accurate 
  # count of the time it took for each classifier
  s = time.time()
  for i in range(iterations):
    #
    # train the classifier on the training data / labels:
    #
    model.fit(X_train, y_train) 
    
  #print ("{0} Iterations Training Time: ".format(iterations), time.time() - s)
  print(f"{iterations} Iterations Training Time: {time.time() - s:.3f}")
  scoreBch = 0
  s = time.time()
  for i in range(iterations):
    #
    # score the classifier on the testing data / labels:
    #
    scoreBch = model.score(X_test, y_test)
    
  #print ("{0} Iterations Scoring Time: ".format(iterations), time.time() - s)
  print(f"{iterations} Iterations Scoring Time: {time.time() - s:.3f}")
  print ("High-Dimensionality Score: ", round((scoreBch*100), 3))
Explanation: Utility function: benchmark times
End of explanation
#
# Create an KNeighbors classifier 
#
from sklearn.neighbors import KNeighborsClassifier
knn = KNeighborsClassifier(n_neighbors=5)
Explanation: Train the Knn classifier
End of explanation
benchmark(knn, X_train, X_test, y_train, y_test, 'KNeighbors')
drawPlots(knn, X_train, X_test, y_train, y_test, 'KNeighbors')
Explanation: And get its benchmark:
End of explanation
#
# Create an SVM classifier
# Use a linear kernel, and set the C value to C (see initial parameters)
#
from sklearn.svm import SVC
svc = SVC(kernel='linear', C=C)
benchmark(svc, X_train, X_test, y_train, y_test, 'SVC')
drawPlots(svc, X_train, X_test, y_train, y_test, 'SVC')
Explanation: Train the SVM Classifier
End of explanation
<END_TASK>