<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 Introduction to Machine Learning
How manual coding works
Opposing basic Idea of Supervised Machine Learning
Hope
Step1:  One of the  Classics
Step2:  https
Step3:  Scatterplot (red=Setosa, green=Versicolour, blue=Virginica)
Step4:  Now for training
Issue
Step5:  We use KNN Classifier with two random features
http
Step6:  Random Feature Selection
Step7:  Training works very fast as we just record all positions in 2d space, no abstraction takes place with KNN
Step8:  Getting some intuition
Step9:  And we also try a completely made up data point
Step10:  Evaluation
Step11:  Scores ok for training, not so good for test
Step12:  We need to smoothen our boundaries
k=10 neighbors
Step13:  Scores for both sets equally low
Step14:  Relationship Overfittung / Underfitting
Black Dots
Step15:  Petal Features seem to be a better idea, what features we take seems to matter
All 4 Features should give best results, though
Step16:  With one neighbor we overfit, let's try more neighbors
<ASSISTANT_TASK:>
Python Code:
import warnings
warnings.filterwarnings('ignore')
%matplotlib inline
%pylab inline
import matplotlib.pylab as plt
import numpy as np
from distutils.version import StrictVersion
import sklearn
print(sklearn.__version__)
assert StrictVersion(sklearn.__version__ ) >= StrictVersion('0.18.1')
# Evtl. hat Azure nur 0.19, wir brauchen aber .20 für das Plotting, dann das hier installieren und Notebook neu starten
# !conda update pandas -y
import pandas as pd
print(pd.__version__)
assert StrictVersion(pd.__version__) >= StrictVersion('0.20.0')
Explanation: Introduction to Machine Learning
How manual coding works
Opposing basic Idea of Supervised Machine Learning
Hope: System can generalize to previously unknown data and situations
Common Use Case: Classification
End of explanation
from sklearn.datasets import load_iris
iris = load_iris()
print(iris.DESCR)
Explanation: One of the  Classics: Classify Iris Type by sizes of their flower
First we load the data set and get an impression
https://en.wikipedia.org/wiki/Iris_flower_data_set
End of explanation
X = iris.data
y = iris.target
X.shape, y.shape
X[0]
y[0]
X_sepal_length = X[:, 0]
X_sepal_width =  X[:, 1]
X_petal_length = X[:, 2]
X_petal_width = X[:, 3]
X_petal_width.shape
Explanation: https://en.wikipedia.org/wiki/File:Petal-sepal.jpg
End of explanation
import matplotlib.pyplot as plt
from matplotlib.colors import ListedColormap
iris_df = pd.DataFrame(iris.data, columns=iris.feature_names)
CMAP = ListedColormap(['#FF0000', '#00FF00', '#0000FF'])
pd.plotting.scatter_matrix(iris_df, c=iris.target, edgecolor='black', figsize=(15, 15), cmap=CMAP)
plt.show()
Explanation: Scatterplot (red=Setosa, green=Versicolour, blue=Virginica)
End of explanation
from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.4, random_state=42, stratify=y)
X_train.shape, y_train.shape, X_test.shape, y_test.shape
Explanation: Now for training
Issue: How do we know if we have trained well?
Splitting data in trainining and (60%) und test (40%)
http://scikit-learn.org/stable/modules/cross_validation.html
End of explanation
from sklearn import neighbors
# ignore this, it is just technical code
# should come from a lib, consider it to appear magically 
# http://scikit-learn.org/stable/auto_examples/neighbors/plot_classification.html
import matplotlib.pyplot as plt
from matplotlib.colors import ListedColormap
cmap_bold = ListedColormap(['#FF0000', '#00FF00', '#0000FF'])
cmap_light = ListedColormap(['#FFAAAA', '#AAFFAA', '#AAAAFF'])
font_size=25
def meshGrid(x_data, y_data):
    h = .02  # step size in the mesh
    x_min, x_max = x_data.min() - 1, x_data.max() + 1
    y_min, y_max = y_data.min() - 1, y_data.max() + 1
    xx, yy = np.meshgrid(np.arange(x_min, x_max, h),
                         np.arange(y_min, y_max, h))
    return (xx,yy)
    
def plotPrediction(clf, x_data, y_data, x_label, y_label, colors, title="", mesh=True):
    xx,yy = meshGrid(x_data, y_data)
    Z = clf.predict(np.c_[xx.ravel(), yy.ravel()])
    # Put the result into a color plot
    Z = Z.reshape(xx.shape)
    plt.figure(figsize=(20,10))
    if mesh:
        plt.pcolormesh(xx, yy, Z, cmap=cmap_light)
    plt.xlim(xx.min(), xx.max())
    plt.ylim(yy.min(), yy.max())
    plt.scatter(x_data, y_data, c=colors, cmap=cmap_bold, s=80, marker='o')
    plt.xlabel(x_label, fontsize=font_size)
    plt.ylabel(y_label, fontsize=font_size)
    plt.title(title, fontsize=font_size)
Explanation: We use KNN Classifier with two random features
http://scikit-learn.org/stable/modules/neighbors.html#classification
End of explanation
X_train_sepal_only = X_train[:, :2]
X_test_sepal_only = X_test[:, :2]
X_train_sepal_only[0]
X_train[0]
Explanation: Random Feature Selection: Sepal Features
End of explanation
clf_sepal = neighbors.KNeighborsClassifier(1)
%time clf_sepal.fit(X_train_sepal_only, y_train)
plotPrediction(clf_sepal, X_train_sepal_only[:, 0], X_train_sepal_only[:, 1], 
               'Sepal length', 'Sepal width', y_train, mesh=False,
                title="Train Data for Sepal Features")
Explanation: Training works very fast as we just record all positions in 2d space, no abstraction takes place with KNN
End of explanation
# 8 is tough because it is directly between 1 and 2
sample_id = 8
# sample_id = 50
sample_feature = X_test_sepal_only[sample_id]
sample_label = y_test[sample_id]
sample_feature
sample_label
clf_sepal.predict([sample_feature])
Explanation: Getting some intuition: what does KNN predict for a previous unknown data set?
End of explanation
clf_sepal.predict([[6.0, 4.5]]) # slightly different from above, still gives 0
Explanation: And we also try a completely made up data point
End of explanation
# clf_sepal.score?
clf_sepal.score(X_train_sepal_only, y_train)
clf_sepal.score(X_test_sepal_only, y_test)
Explanation: Evaluation: What perecentage in each set is corretly predicted?
End of explanation
plotPrediction(clf_sepal, X_train_sepal_only[:, 0], X_train_sepal_only[:, 1], 
               'Sepal length', 'Sepal width', y_train,
               title="Highly Fragmented Decision Boundaries for Train Data")
plotPrediction(clf_sepal, X_test_sepal_only[:, 0], X_test_sepal_only[:, 1],
               'Sepal length', 'Sepal width', y_test,
               title="Same Decision Boundaries don't work well for Test Data")
Explanation: Scores ok for training, not so good for test: Overfitting
To understand what has happened we draw Decision Boundaries
For each data point we draw the prediction our model would give as an area
End of explanation
# neighbors.KNeighborsClassifier?
clf_sepal_10 = neighbors.KNeighborsClassifier(10)
clf_sepal_10.fit(X_train_sepal_only, y_train)
clf_sepal_10.score(X_train_sepal_only, y_train)
clf_sepal_10.score(X_test_sepal_only, y_test)
Explanation: We need to smoothen our boundaries
k=10 neighbors
End of explanation
plotPrediction(clf_sepal_10, X_train_sepal_only[:, 0], X_train_sepal_only[:, 1], 
               'Sepal length', 'Sepal width', y_train,
               title="Model too simple even for Train Data")
plotPrediction(clf_sepal_10, X_test_sepal_only[:, 0], X_test_sepal_only[:, 1], 
               'Sepal length', 'Sepal width', y_test,
               title="Model also too simple for Test Data")
Explanation: Scores for both sets equally low: Undefitting
End of explanation
X_train_petal_only = X_train[:, 2:]
X_test_petal_only = X_test[:, 2:]
X_train_petal_only[0]
X_train[0]
clf_petal_10 = neighbors.KNeighborsClassifier(10)
clf_petal_10.fit(X_train_petal_only, y_train)
plotPrediction(clf_petal_10, X_train_petal_only[:, 0], X_train_petal_only[:, 1], 
               'Petal length', 'Petal width', y_train,
               title="Simple model looks good for Train Data")
plotPrediction(clf_petal_10, X_test_petal_only[:, 0], X_test_petal_only[:, 1], 
               'Petal length', 'Petal width', y_test,
               title="Simple model looks good even for Test Data")
clf_petal_10.score(X_train_petal_only, y_train)
clf_petal_10.score(X_test_petal_only, y_test)
Explanation: Relationship Overfittung / Underfitting
Black Dots: Training Data
Gray Dots: Test Data
Feature Selektion
Sepal Features seem to always either over- or underfit
Let us try Petal Features
End of explanation
clf = neighbors.KNeighborsClassifier(1)
clf.fit(X_train, y_train)
clf.score(X_train, y_train)
clf.score(X_test, y_test)
Explanation: Petal Features seem to be a better idea, what features we take seems to matter
All 4 Features should give best results, though
End of explanation
clf = neighbors.KNeighborsClassifier(13)
clf.fit(X_train, y_train)
clf.score(X_train, y_train)
clf.score(X_test, y_test)
Explanation: With one neighbor we overfit, let's try more neighbors
End of explanation
<END_TASK>