<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 Cross-Validation and scoring methods
To evaluate how well our supervised models generalize, so far we split our data into a training and a test set
Step1:  The labels in iris are sorted, which means that if we split the data as illustrated above, the first fold will only have the label 0 in it, while the last one will only have the label 2
Step2:  To avoid this problem in evaluation, we first shuffle our data
Step3:  Now implementing cross-validation is easy
Step4:  Let's check that our test mask does the right thing
Step5:  And now let's look a the scores we computed
Step6:  As you can see, there is a rather wide spectrum of scores from 90% correct to 100% correct. If we only did a single split, we might have gotten either answer.
As cross-validation is such a common pattern in machine learning, there are functions to do the above for you with much more flexibility and less code.
The sklearn.cross_validation module has all functions related to cross validation. There easiest function is cross_val_score which takes an estimator and a dataset, and will do all of the splitting for you
Step7:  As you can see, the function uses three folds by default. You can change the number of folds using the cv argument
Step8:  There are also helper objects in the cross-validation module that will generate indices for you for all kinds of different cross-validation methods, including k-fold
Step9:  By default, cross_val_score will use StrafifiedKFold for classification, which ensures that the class proportions in the dataset are reflected in each fold. If you have a binary classification dataset with 90% of data point belonging to class 0, that would mean that in each fold, 90% of datapoints would belong to class 0.
If you would just use KFold cross-validation, it is likely that you would generate a split that only contains class 0.
It is generally a good idea to use StratifiedKFold whenever you do classification.
StratifiedKFold would also remove our need to shuffle iris.
Let's see what kinds of folds it generates on the unshuffled iris dataset.
Each cross-validation class is a generator of sets of training and test indices
Step10:  As you can see, there are a couple of samples from the beginning, then from the middle, and then from the end, in each of the folds.
This way, the class ratios are preserved. Let's visualize the split
Step11:  For comparison, again the standard KFold, that ignores the labels
Step12:  Keep in mind that increasing the number of folds will give you a larger training dataset, but will lead to more repetitions, and therefore a slower evaluation
Step13:  Another helpful cross-validation generator is ShuffleSplit. This generator simply splits of a random portion of the data repeatedly. This allows the user to specify the number of repetitions and the training set size independently
Step14:  If you want a more robust estimate, you can just increase the number of iterations
Step15:  You can use all of these cross-validation generators with the cross_val_score method
<ASSISTANT_TASK:>
Python Code:
from sklearn.datasets import load_iris
from sklearn.neighbors import KNeighborsClassifier
iris = load_iris()
X, y = iris.data, iris.target
classifier = KNeighborsClassifier()
Explanation: Cross-Validation and scoring methods
To evaluate how well our supervised models generalize, so far we split our data into a training and a test set:
However, often (labeled) data is precious, and this approach lets us only use ~ 3/4 of our data for training. On the other hand, we will only ever try to apply our model 1/4 of our data for testing.
A common way to use more of the data to built a model, but also get a more robust estimate of the generalization performance is cross-validation.
In cross-validation, the data is split repeatedly into a training and test-set, with a separate model built for every pair. The test-set scores are then aggregated for a more robust estimate.
The most common way to do cross-validation is k-fold cross-validation, in which the data is first split into k (often 5 or 10) equal-sized folds, and then for each iteration, one of the k folds is used as test data, and the rest as training data:
This way, each data point will be in the test-set exactly once, and we can use all but a k'th of the data for training.
Let us apply this technique to evaluate the KNeighborsClassifier algorithm on the Iris dataset:
End of explanation
y
Explanation: The labels in iris are sorted, which means that if we split the data as illustrated above, the first fold will only have the label 0 in it, while the last one will only have the label 2:
End of explanation
import numpy as np
rng = np.random.RandomState(0)
permutation = rng.permutation(len(X))
X, y = X[permutation], y[permutation]
print(y)
Explanation: To avoid this problem in evaluation, we first shuffle our data:
End of explanation
k = 5
n_samples = len(X)
fold_size = n_samples // k
scores = []
masks = []
for fold in range(k):
    # generate a boolean mask for the test set in this fold
    test_mask = np.zeros(n_samples, dtype=bool)
    test_mask[fold * fold_size : (fold + 1) * fold_size] = True
    # store the mask for visualization
    masks.append(test_mask)
    # create training and test sets using this mask
    X_test, y_test = X[test_mask], y[test_mask]
    X_train, y_train = X[~test_mask], y[~test_mask]
    # fit the classifier
    classifier.fit(X_train, y_train)
    # compute the score and record it
    scores.append(classifier.score(X_test, y_test))
Explanation: Now implementing cross-validation is easy:
End of explanation
import matplotlib.pyplot as plt
%matplotlib inline
plt.matshow(masks)
Explanation: Let's check that our test mask does the right thing:
End of explanation
print(scores)
print(np.mean(scores))
Explanation: And now let's look a the scores we computed:
End of explanation
from sklearn.cross_validation import cross_val_score
scores = cross_val_score(classifier, X, y)
print(scores)
print(np.mean(scores))
Explanation: As you can see, there is a rather wide spectrum of scores from 90% correct to 100% correct. If we only did a single split, we might have gotten either answer.
As cross-validation is such a common pattern in machine learning, there are functions to do the above for you with much more flexibility and less code.
The sklearn.cross_validation module has all functions related to cross validation. There easiest function is cross_val_score which takes an estimator and a dataset, and will do all of the splitting for you:
End of explanation
cross_val_score(classifier, X, y, cv=5)
Explanation: As you can see, the function uses three folds by default. You can change the number of folds using the cv argument:
End of explanation
from sklearn.cross_validation import KFold, StratifiedKFold, ShuffleSplit, LeavePLabelOut
Explanation: There are also helper objects in the cross-validation module that will generate indices for you for all kinds of different cross-validation methods, including k-fold:
End of explanation
cv = StratifiedKFold(iris.target, n_folds=5)
for train, test in cv:
    print(test)
Explanation: By default, cross_val_score will use StrafifiedKFold for classification, which ensures that the class proportions in the dataset are reflected in each fold. If you have a binary classification dataset with 90% of data point belonging to class 0, that would mean that in each fold, 90% of datapoints would belong to class 0.
If you would just use KFold cross-validation, it is likely that you would generate a split that only contains class 0.
It is generally a good idea to use StratifiedKFold whenever you do classification.
StratifiedKFold would also remove our need to shuffle iris.
Let's see what kinds of folds it generates on the unshuffled iris dataset.
Each cross-validation class is a generator of sets of training and test indices:
End of explanation
def plot_cv(cv, n_samples):
    masks = []
    for train, test in cv:
        mask = np.zeros(n_samples, dtype=bool)
        mask[test] = 1
        masks.append(mask)
    
    plt.matshow(masks)
plot_cv(StratifiedKFold(iris.target, n_folds=5), len(iris.target))
Explanation: As you can see, there are a couple of samples from the beginning, then from the middle, and then from the end, in each of the folds.
This way, the class ratios are preserved. Let's visualize the split:
End of explanation
plot_cv(KFold(len(iris.target), n_folds=5), len(iris.target))
Explanation: For comparison, again the standard KFold, that ignores the labels:
End of explanation
plot_cv(KFold(len(iris.target), n_folds=10), len(iris.target))
Explanation: Keep in mind that increasing the number of folds will give you a larger training dataset, but will lead to more repetitions, and therefore a slower evaluation:
End of explanation
plot_cv(ShuffleSplit(len(iris.target), n_iter=5, test_size=.2), len(iris.target))
Explanation: Another helpful cross-validation generator is ShuffleSplit. This generator simply splits of a random portion of the data repeatedly. This allows the user to specify the number of repetitions and the training set size independently:
End of explanation
plot_cv(ShuffleSplit(len(iris.target), n_iter=20, test_size=.2), len(iris.target))
Explanation: If you want a more robust estimate, you can just increase the number of iterations:
End of explanation
cv = ShuffleSplit(len(iris.target), n_iter=5, test_size=.2)
cross_val_score(classifier, X, y, cv=cv)
Explanation: You can use all of these cross-validation generators with the cross_val_score method:
End of explanation
<END_TASK>