<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
Step1:  Now let's look at some classification methods. 
Nearest Neighbor
Step2:  Exercise
Step3:  Exercise
Step4:  Now write a loop that does this using 100 different randomly generated datasets, and plot the mean across datasets.  This will take a couple of minutes to run.
Step5:  Linear discriminant analysis
Step6:  Logistic regression
Step7:  Support vector machines
Step8:  Exercise
Step9:  Exercise
Step10:  Plot the boundary for the classifier with the best performance
<ASSISTANT_TASK:>
Python Code:
# adapted from http://scikit-learn.org/stable/auto_examples/neighbors/plot_classification.html#example-neighbors-plot-classification-py
n_neighbors = 30
 # step size in the mesh
# Create color maps
cmap_light = ListedColormap(['#FFAAAA', '#AAFFAA'])
cmap_bold = ListedColormap(['#FF0000', '#00FF00'])
clf = sklearn.neighbors.KNeighborsClassifier(n_neighbors, weights='uniform')
clf.fit(d, cl)
def plot_cls_with_decision_surface(d,cl,clf,h = .25 ):
    Plot the decision boundary. For that, we will assign a color to each
    point in the mesh [x_min, m_max]x[y_min, y_max].
    h= step size in the grid
    
    x_min, x_max = d[:, 0].min() - 1, d[:, 0].max() + 1
    y_min, y_max = d[:, 1].min() - 1, d[:, 1].max() + 1
    xx, yy = numpy.meshgrid(numpy.arange(x_min, x_max, h),
                         numpy.arange(y_min, y_max, h))
    Z = clf.predict(numpy.c_[xx.ravel(), yy.ravel()])
    # Put the result into a color plot
    Z = Z.reshape(xx.shape)
    plt.pcolormesh(xx, yy, Z, cmap=cmap_light)
    # Plot also the training points
    plt.scatter(d[:, 0], d[:, 1], c=cl, cmap=cmap_bold)
    plt.xlim(xx.min(), xx.max())
    plt.ylim(yy.min(), yy.max())
plot_cls_with_decision_surface(d,cl,clf)
Explanation: Now let's look at some classification methods. 
Nearest Neighbor
End of explanation
def classify(d,cl,clf,cv):
    pred=numpy.zeros(n)
    for train,test in cv:
        clf.fit(d[train,:],cl[train])
        pred[test]=clf.predict(d[test,:])
    return sklearn.metrics.accuracy_score(cl,pred),sklearn.metrics.confusion_matrix(cl,pred)
clf=sklearn.neighbors.KNeighborsClassifier(n_neighbors, weights='uniform')
# use stratified k-fold crossvalidation, which keeps the proportion of classes roughly
# equal across folds
cv=sklearn.cross_validation.StratifiedKFold(cl, 8)
acc,confusion=classify(d,cl,clf,cv)
print acc
print confusion
Explanation: Exercise: Change the number of nearest neighbors and see how it changes the surface.
Now let's write a function to perform cross-validation and compute prediction accuracy.
End of explanation
accuracy_knn=numpy.zeros(30)
for i in range(1,31):
    clf=sklearn.neighbors.KNeighborsClassifier(i, weights='uniform')
    accuracy_knn[i-1],_=classify(d,cl,clf,cv)
plt.plot(range(1,31),accuracy_knn)
Explanation: Exercise:  Loop through different levels of n_neighbors (from 1 to 30) and compute the accuracy.
End of explanation
accuracy_knn=numpy.zeros((100,30))
for x in range(100):
    ds_cl,ds_x=make_class_data(multiplier=[1.1,1.1],N=n)
    ds_cv=sklearn.cross_validation.StratifiedKFold(ds_cl, 8)
    for i in range(1,31):
        clf=sklearn.neighbors.KNeighborsClassifier(i, weights='uniform')
        accuracy_knn[x,i-1],_=classify(ds_x,ds_cl,clf,ds_cv)
plt.plot(range(1,31),numpy.mean(accuracy_knn,0))
plt.xlabel('number of nearest neighbors')
plt.ylabel('accuracy')
Explanation: Now write a loop that does this using 100 different randomly generated datasets, and plot the mean across datasets.  This will take a couple of minutes to run.
End of explanation
clf=sklearn.lda.LDA()
cv=sklearn.cross_validation.LeaveOneOut(n)
acc,confusion=classify(d,cl,clf,cv)
print acc
print confusion
plot_cls_with_decision_surface(d,cl,clf)
Explanation: Linear discriminant analysis
End of explanation
clf=sklearn.linear_model.LogisticRegression(C=0.5)
acc,confusion=classify(d,cl,clf,cv)
print acc
print confusion
plot_cls_with_decision_surface(d,cl,clf)
Explanation: Logistic regression
End of explanation
clf=sklearn.svm.SVC(kernel='linear')
acc,confusion=classify(d,cl,clf,cv)
print acc
print confusion
plot_cls_with_decision_surface(d,cl,clf)
Explanation: Support vector machines
End of explanation
clf=sklearn.svm.SVC(kernel='rbf')
acc,confusion=classify(d,cl,clf,cv)
print acc
print confusion
plot_cls_with_decision_surface(d,cl,clf)
Explanation: Exercise: Implement the example above using a nonlinear SVM with a radial basis kernel.
End of explanation
gammavals=numpy.arange(0.0,0.2,0.01)
accuracy_rbf=numpy.zeros(len(gammavals))
for i in range(len(gammavals)):
    clf=sklearn.svm.SVC(kernel='rbf',gamma=gammavals[i])
    accuracy_rbf[i],_=classify(d,cl,clf,cv)
plt.plot(gammavals,accuracy_rbf)
plt.xlabel('gamma')
plt.ylabel('accuracy')
Explanation: Exercise: Try the RBF SVC using a range of values for the gamma parameter.
NOTE: For real data analysis, you cannot determine the best value of gamma this way, because you would be peeking at the test data which will make your results overly optimistic.  Instead, you would need to use nested cross-validation loops; we will come to this later
End of explanation
maxgamma=gammavals[numpy.where(accuracy_rbf==numpy.max(accuracy_rbf))]
if len(maxgamma)>1:
    maxgamma=maxgamma[0]
print 'Best gamma:', maxgamma
clf=sklearn.svm.SVC(kernel='rbf',gamma=maxgamma)
acc,_=classify(d,cl,clf,cv)
print 'Accuracy:',acc
plot_cls_with_decision_surface(d,cl,clf)
Explanation: Plot the boundary for the classifier with the best performance
End of explanation
<END_TASK>