<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 The issues associated with validation and 
cross-validation are some of the most important
aspects of the practice of machine learning.  Selecting the optimal model
for your data is vital, and is a piece of the problem that is not often
appreciated by machine learning practitioners.
Of core importance is the following question
Step1:  Learning Curves
What the right model for a dataset is depends critically on how much data we have. More data allows us to be more confident about building a complex model. Lets built some intuition on why that is. Look at the following datasets
Step2:  They all come from the same underlying process. But if you were asked to make a prediction, you would be more likely to draw a straight line for the left-most one, as there are only very few datapoints, and no real rule is apparent. For the dataset in the middle, some structure is recognizable, though the exact shape of the true function is maybe not obvious. With even more data on the right hand side, you would probably be very comfortable with drawing a curved line with a lot of certainty.
A great way to explore how a model fit evolves with different dataset sizes are learning curves.
A learning curve plots the validation error for a given model against different training set sizes.
But first, take a moment to think about what we're going to see
Step3:  You can see that for the model with kernel = linear, the validation score doesn't really decrease as more data is given.
Notice that the validation error generally decreases with a growing training set,
while the training error generally increases with a growing training set.  From
this we can infer that as the training size increases, they will converge to a single
value.
From the above discussion, we know that kernel = linear
underfits the data. This is indicated by the fact that both the
training and validation errors are very high. When confronted with this type of learning curve,
we can expect that adding more training data will not help matters
<ASSISTANT_TASK:>
Python Code:
import numpy as np
import matplotlib.pyplot as plt
from sklearn.pipeline import Pipeline
from sklearn.svm import SVR
from sklearn import cross_validation
np.random.seed(0)
n_samples = 200
kernels = ['linear', 'poly', 'rbf']
true_fun = lambda X: X ** 3
X = np.sort(5 * (np.random.rand(n_samples) - .5))
y = true_fun(X) + .01 * np.random.randn(n_samples)
plt.figure(figsize=(14, 5))
for i in range(len(kernels)):
    ax = plt.subplot(1, len(kernels), i + 1)
    plt.setp(ax, xticks=(), yticks=())
    model = SVR(kernel=kernels[i], C=5)
    model.fit(X[:, np.newaxis], y)
    # Evaluate the models using crossvalidation
    scores = cross_validation.cross_val_score(model,
        X[:, np.newaxis], y, scoring="mean_squared_error", cv=10)
    X_test = np.linspace(3 * -.5, 3 * .5, 100)
    plt.plot(X_test, model.predict(X_test[:, np.newaxis]), label="Model")
    plt.plot(X_test, true_fun(X_test), label="True function")
    plt.scatter(X, y, label="Samples")
    plt.xlabel("x")
    plt.ylabel("y")
    plt.xlim((-3 * .5, 3 * .5))
    plt.ylim((-1, 1))
    plt.legend(loc="best")
    plt.title("Kernel {}\nMSE = {:.2e}(+/- {:.2e})".format(
        kernels[i], -scores.mean(), scores.std()))
plt.show()
Explanation: The issues associated with validation and 
cross-validation are some of the most important
aspects of the practice of machine learning.  Selecting the optimal model
for your data is vital, and is a piece of the problem that is not often
appreciated by machine learning practitioners.
Of core importance is the following question:
If our estimator is underperforming, how should we move forward?
Use simpler or more complicated model?
Add more features to each observed data point?
Add more training samples?
The answer is often counter-intuitive.  In particular, sometimes using a
more complicated model will give worse results.  Also, sometimes adding
training data will not improve your results.  The ability to determine
what steps will improve your model is what separates the successful machine
learning practitioners from the unsuccessful.
Learning Curves and Validation Curves
One way to address this issue is to use what are often called Learning Curves.
Given a particular dataset and a model we'd like to fit (e.g. using feature creation and linear regression), we'd
like to tune our value of the hyperparameter kernel to give us the best fit. We can visualize the different regimes with the following plot, modified from the sklearn examples here
End of explanation
import numpy as np
import matplotlib.pyplot as plt
from sklearn import cross_validation
np.random.seed(0)
n_samples = 200
true_fun = lambda X: X ** 3
X = np.sort(5 * (np.random.rand(n_samples) - .5))
y = true_fun(X) + .02 * np.random.randn(n_samples)
X = X[:, None]
y = y
f, axarr = plt.subplots(1, 3)
axarr[0].scatter(X[::20], y[::20])
axarr[0].set_xlim((-3 * .5, 3 * .5))
axarr[0].set_ylim((-1, 1))
axarr[1].scatter(X[::10], y[::10])
axarr[1].set_xlim((-3 * .5, 3 * .5))
axarr[1].set_ylim((-1, 1))
axarr[2].scatter(X, y)
axarr[2].set_xlim((-3 * .5, 3 * .5))
axarr[2].set_ylim((-1, 1))
plt.show()
Explanation: Learning Curves
What the right model for a dataset is depends critically on how much data we have. More data allows us to be more confident about building a complex model. Lets built some intuition on why that is. Look at the following datasets:
End of explanation
from sklearn.learning_curve import learning_curve
from sklearn.svm import SVR
training_sizes, train_scores, test_scores = learning_curve(SVR(kernel='linear'), X, y, cv=10, scoring="mean_squared_error",
                                                           train_sizes=[.6, .7, .8, .9, 1.])
# Use the negative because we want to minimize squared error
plt.plot(training_sizes, -train_scores.mean(axis=1), label="training scores")
plt.plot(training_sizes, -test_scores.mean(axis=1), label="test scores")
plt.ylim((0, 50))
plt.legend(loc='best')
Explanation: They all come from the same underlying process. But if you were asked to make a prediction, you would be more likely to draw a straight line for the left-most one, as there are only very few datapoints, and no real rule is apparent. For the dataset in the middle, some structure is recognizable, though the exact shape of the true function is maybe not obvious. With even more data on the right hand side, you would probably be very comfortable with drawing a curved line with a lot of certainty.
A great way to explore how a model fit evolves with different dataset sizes are learning curves.
A learning curve plots the validation error for a given model against different training set sizes.
But first, take a moment to think about what we're going to see:
Questions:
As the number of training samples are increased, what do you expect to see for the training error?  For the validation error?
Would you expect the training error to be higher or lower than the validation error?  Would you ever expect this to change?
We can run the following code to plot the learning curve for a kernel = linear model:
End of explanation
from sklearn.learning_curve import learning_curve
from sklearn.svm import SVR
training_sizes, train_scores, test_scores = learning_curve(SVR(kernel='rbf'), X, y, cv=10, scoring="mean_squared_error",
                                                           train_sizes=[.6, .7, .8, .9, 1.])
# Use the negative because we want to minimize squared error
plt.plot(training_sizes, -train_scores.mean(axis=1), label="training scores")
plt.plot(training_sizes, -test_scores.mean(axis=1), label="test scores")
plt.ylim((0, 50))
plt.legend(loc='best')
Explanation: You can see that for the model with kernel = linear, the validation score doesn't really decrease as more data is given.
Notice that the validation error generally decreases with a growing training set,
while the training error generally increases with a growing training set.  From
this we can infer that as the training size increases, they will converge to a single
value.
From the above discussion, we know that kernel = linear
underfits the data. This is indicated by the fact that both the
training and validation errors are very high. When confronted with this type of learning curve,
we can expect that adding more training data will not help matters: both
lines will converge to a relatively high error.
When the learning curves have converged to a high error, we have an underfitting model.
An underfitting model can be improved by:
Using a more sophisticated model (i.e. in this case, increase complexity of the kernel parameter)
Gather more features for each sample.
Decrease regularization in a regularized model.
A underfitting model cannot be improved, however, by increasing the number of training
samples (do you see why?)
Now let's look at an overfit model:
End of explanation
<END_TASK>