<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 Supervised Learning Part 1 -- Classification
To visualize the workings of machine learning algorithms, it is often helpful to study two-dimensional or one-dimensional data, that is data with only one or two features. While in practice, datasets usually have many more features, it is hard to plot high-dimensional data in on two-dimensional screens.
We will illustrate some very simple examples before we move on to more "real world" data sets.
First, we will look at a two class classification problem in two dimensions. We use the synthetic data generated by the make_blobs function.
Step1:  As the data is two-dimensional, we can plot each sample as a point in a two-dimensional coordinate system, with the first feature being the x-axis and the second feature being the y-axis.
Step2:  Classification is a supervised task, and since we are interested in its performance on unseen data, we split our data into two parts
Step3:  The scikit-learn estimator API
<img src="figures/supervised_workflow.svg" width="100%">
Every algorithm is exposed in scikit-learn via an ''Estimator'' object. (All models in scikit-learn have a very consistent interface). For instance, we first import the logistic regression class.
Step4:  Next, we instantiate the estimator object.
Step5:  To built the model from our data, that is to learn how to classify new points, we call the fit function with the training data, and the corresponding training labels (the desired output for the training data point)
Step6:  (Some estimator methods such as fit return self by default. Thus, after executing the code snippet above, you will see the default parameters of this particular instance of LogisticRegression. Another way of retrieving the estimator's ininitialization parameters is to execute classifier.get_params(), which returns a parameter dictionary.)
We can then apply the model to unseen data and use the model to predict the estimated outcome using the predict method
Step7:  We can compare these against the true labels
Step8:  We can evaluate our classifier quantitatively by measuring what fraction of predictions is correct. This is called accuracy
Step9:  There is also a convenience function , score, that all scikit-learn classifiers have to compute this directly from the test data
Step10:  It is often helpful to compare the generalization performance (on the test set) to the performance on the training set
Step11:  LogisticRegression is a so-called linear model,
that means it will create a decision that is linear in the input space. In 2d, this simply means it finds a line to separate the blue from the red
Step12:  Estimated parameters
Step13:  Another classifier
Step14:  This time we set a parameter of the KNeighborsClassifier to tell it we only want to look at one nearest neighbor
Step15:  We fit the model with out training data
Step16:  <div class="alert alert-success">
    <b>EXERCISE</b>
<ASSISTANT_TASK:>
Python Code:
from sklearn.datasets import make_blobs
X, y = make_blobs(centers=2, random_state=0)
print('X ~ n_samples x n_features:', X.shape)
print('y ~ n_samples:', y.shape)
print('\nFirst 5 samples:\n', X[:5, :])
print('\nFirst 5 labels:', y[:5])
Explanation: Supervised Learning Part 1 -- Classification
To visualize the workings of machine learning algorithms, it is often helpful to study two-dimensional or one-dimensional data, that is data with only one or two features. While in practice, datasets usually have many more features, it is hard to plot high-dimensional data in on two-dimensional screens.
We will illustrate some very simple examples before we move on to more "real world" data sets.
First, we will look at a two class classification problem in two dimensions. We use the synthetic data generated by the make_blobs function.
End of explanation
plt.scatter(X[y == 0, 0], X[y == 0, 1], 
            c='blue', s=40, label='0')
plt.scatter(X[y == 1, 0], X[y == 1, 1], 
            c='red', s=40, label='1', marker='s')
plt.xlabel('first feature')
plt.ylabel('second feature')
plt.legend(loc='upper right');
Explanation: As the data is two-dimensional, we can plot each sample as a point in a two-dimensional coordinate system, with the first feature being the x-axis and the second feature being the y-axis.
End of explanation
from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(X, y,
                                                    test_size=0.25,
                                                    random_state=1234,
                                                    stratify=y)
Explanation: Classification is a supervised task, and since we are interested in its performance on unseen data, we split our data into two parts:
a training set that the learning algorithm uses to fit the model
a test set to evaluate the generalization performance of the model
The train_test_split function from the model_selection module does that for us -- we will use it to split a dataset into 75% training data and 25% test data.
<img src="figures/train_test_split_matrix.svg" width="100%">
End of explanation
from sklearn.linear_model import LogisticRegression
Explanation: The scikit-learn estimator API
<img src="figures/supervised_workflow.svg" width="100%">
Every algorithm is exposed in scikit-learn via an ''Estimator'' object. (All models in scikit-learn have a very consistent interface). For instance, we first import the logistic regression class.
End of explanation
classifier = LogisticRegression()
X_train.shape
y_train.shape
Explanation: Next, we instantiate the estimator object.
End of explanation
classifier.fit(X_train, y_train)
Explanation: To built the model from our data, that is to learn how to classify new points, we call the fit function with the training data, and the corresponding training labels (the desired output for the training data point):
End of explanation
prediction = classifier.predict(X_test)
Explanation: (Some estimator methods such as fit return self by default. Thus, after executing the code snippet above, you will see the default parameters of this particular instance of LogisticRegression. Another way of retrieving the estimator's ininitialization parameters is to execute classifier.get_params(), which returns a parameter dictionary.)
We can then apply the model to unseen data and use the model to predict the estimated outcome using the predict method:
End of explanation
print(prediction)
print(y_test)
Explanation: We can compare these against the true labels:
End of explanation
np.mean(prediction == y_test)
Explanation: We can evaluate our classifier quantitatively by measuring what fraction of predictions is correct. This is called accuracy:
End of explanation
classifier.score(X_test, y_test)
Explanation: There is also a convenience function , score, that all scikit-learn classifiers have to compute this directly from the test data:
End of explanation
classifier.score(X_train, y_train)
Explanation: It is often helpful to compare the generalization performance (on the test set) to the performance on the training set:
End of explanation
from figures import plot_2d_separator
plt.scatter(X[y == 0, 0], X[y == 0, 1], 
            c='blue', s=40, label='0')
plt.scatter(X[y == 1, 0], X[y == 1, 1], 
            c='red', s=40, label='1', marker='s')
plt.xlabel("first feature")
plt.ylabel("second feature")
plot_2d_separator(classifier, X)
plt.legend(loc='upper right');
Explanation: LogisticRegression is a so-called linear model,
that means it will create a decision that is linear in the input space. In 2d, this simply means it finds a line to separate the blue from the red:
End of explanation
print(classifier.coef_)
print(classifier.intercept_)
Explanation: Estimated parameters: All the estimated model parameters are attributes of the estimator object ending by an underscore. Here, these are the coefficients and the offset of the line:
End of explanation
from sklearn.neighbors import KNeighborsClassifier
Explanation: Another classifier: K Nearest Neighbors
Another popular and easy to understand classifier is K nearest neighbors (kNN).  It has one of the simplest learning strategies: given a new, unknown observation, look up in your reference database which ones have the closest features and assign the predominant class.
The interface is exactly the same as for LogisticRegression above.
End of explanation
knn = KNeighborsClassifier(n_neighbors=1)
Explanation: This time we set a parameter of the KNeighborsClassifier to tell it we only want to look at one nearest neighbor:
End of explanation
knn.fit(X_train, y_train)
plt.scatter(X[y == 0, 0], X[y == 0, 1], 
            c='blue', s=40, label='0')
plt.scatter(X[y == 1, 0], X[y == 1, 1], 
            c='red', s=40, label='1', marker='s')
plt.xlabel("first feature")
plt.ylabel("second feature")
plot_2d_separator(knn, X)
plt.legend(loc='upper right');
knn.score(X_test, y_test)
Explanation: We fit the model with out training data
End of explanation
# %load solutions/05A_knn_with_diff_k.py
Explanation: <div class="alert alert-success">
    <b>EXERCISE</b>:
     <ul>
      <li>
      Apply the KNeighborsClassifier to the ``iris`` dataset. Play with different values of the ``n_neighbors`` and observe how training and test score change.
      </li>
    </ul>
</div>
End of explanation
<END_TASK>