<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 ===Cross-validation===
Here we consider a class of methods that estimate the test error (the average error that results from using a statistical learning method to predict the response on a new observation) by holding out a subset of the training observations from the fitting process, and then applying the statistical learning method to those held out observations.
Get the data
We use the dataset auto from the UCI Machine Learning Repository.  
Specifically, we will use the feature horsepower (i.e., the car's HP number) to predict its fuel consumption (the mpg feature).
Step1:  397 examples of 9 features
Step2:  We are interested in the mpg and the horsepower features.
Step3:  There are NaNs. We remove those observations before proceeding.
Step4:  Visualise the data
Step5:  Fit a linear regression model
Step6:  These are the model's coefficients
Step7:  The model would look like $mpg = 39.9 - 0.15 * horsepower$.
Let's display the linear regression line in the previous chart
Step8:  We can already see that a simple linear regression model will not be very accurate ... probably a curve would better approximate the real fuel consumption better than a line.
Let's measure the accuracy using the test-set hold-out approach.
Test-set approach
Here we randomly divide the available set of observations into two parts
Step9:  Measure the accuracy of the model
We fit the model on the train dataset
Step10:  And measure the accuracy on the test dataset.
Note
Step11:  The validation estimate of the test error can be highly variable, depending on precisely which observations are included in the training set and which observations are included in the test set. Every time you split the data into different train/test datasets you get different models and scores. You can try it yourself by removing the fixed random seed above.
Cross-validation approach
A solution to this problem is a procedure called cross-validation (CV for short).  
K-Fold
In the basic approach, called K-fold CV, the training set is split into K smaller sets.
We leave out part k, fit the model to the other K-1 parts (combined), and then obtain predictions for the left-out k-th part.
This is done in turn for each part k = 1,2,...K, and then the results are combined.
Step12:  KFold divides all the samples in K groups of samples, called folds, of equal sizes (if possible).
Step13:  Note
Step14:  Leave One Out (LOO) cross-validation
LOO is a special case of the Kfold splitting approach
Step15:  Note
Step16:  Now the accuracy of 0.606 is a much more reliable measure than the one from the test-set approach, which could change depending on which test set has been randomly selected.
A generic cross-validation score evaluating function.
The sklearn helper cross_val_score can quickly estimate the score of a model, using one CV approach, therefore substituting the above code with a more compact form
Step17:  In the first iteration, the accuracy is 58%, in the second iteration the accuracy is 65% and so on.  
Generally a 10-folds is a good compromise between speed and accuracy.
A model with polynomial features
Now we try to improve our simple model's accuracy by adding polynomial terms to our feature, i.e. transforming it into its power.
Step18:  Then we fit a new model to the transformed data
Step19:  It looks better, so we can try with a cubic function
Step20:  These results show that a model which predicts mpg using a quadratic function of horsepower performs better than a model that involves only a linear function of horsepower, and there is little evidence in favour of a model that uses a cubic function of horsepower.
Using the CV approach to tune parameters (model selection)
We can repeat this procedure for increasingly complex polynomial fits. 
This is called model selection.  
To automate the process, we use a for()  loop, which iteratively fits polynomial regressions for polynomials of order i = 1 to i = 15, computes the associated cross-validation score and stores it.
<ASSISTANT_TASK:>
Python Code:
import pandas as pd
autoData = pd.read_csv('../datasets/Auto.csv', na_values='?') 
autoData.shape
Explanation: ===Cross-validation===
Here we consider a class of methods that estimate the test error (the average error that results from using a statistical learning method to predict the response on a new observation) by holding out a subset of the training observations from the fitting process, and then applying the statistical learning method to those held out observations.
Get the data
We use the dataset auto from the UCI Machine Learning Repository.  
Specifically, we will use the feature horsepower (i.e., the car's HP number) to predict its fuel consumption (the mpg feature).
End of explanation
autoData.columns
Explanation: 397 examples of 9 features
End of explanation
autoData.head(5)
autoData.info()
autoData.isnull().values.any()
Explanation: We are interested in the mpg and the horsepower features.
End of explanation
autoData[autoData.isnull().any(axis=1)]
autoData.dropna(axis=0, inplace=True) # Drop any row that has NaNs within it
autoData[autoData.isnull().any(axis=1)]
  # horsepower feature is our input X
X = autoData[['horsepower']]
X.shape
X.describe()
m = X.shape[0]  # number of examples
m
  # mpg is our output Y, what we want to predict
Y = autoData.mpg.copy() # copy “y” column values out
len(Y)
Explanation: There are NaNs. We remove those observations before proceeding.
End of explanation
import matplotlib.pyplot as plt
%matplotlib inline
fig = plt.scatter(X, Y)
fig = plt.grid(True)
fig = plt.xlabel('Horsepower')
fig = plt.ylabel('Miles per Gallon (mpg)')
Explanation: Visualise the data
End of explanation
from sklearn import linear_model
#
# : Create the linear regression model 
#
model = linear_model.LinearRegression()
model.fit(X, Y)
Explanation: Fit a linear regression model
End of explanation
model.intercept_
model.coef_
Explanation: These are the model's coefficients:
End of explanation
y_plot = model.predict([[40],[230]])
y_plot
fig = plt.scatter(X, Y)
fig = plt.grid(True)
fig = plt.xlabel('Horsepower')
fig = plt.ylabel('Miles per Gallon (mpg)')
fig = plt.plot([40,230], y_plot, '-', color='darkred', linewidth=2)
Explanation: The model would look like $mpg = 39.9 - 0.15 * horsepower$.
Let's display the linear regression line in the previous chart:
End of explanation
from sklearn.model_selection import train_test_split
x_train, x_test, y_train, y_test = train_test_split(X, Y, test_size=0.2, random_state=1)
Explanation: We can already see that a simple linear regression model will not be very accurate ... probably a curve would better approximate the real fuel consumption better than a line.
Let's measure the accuracy using the test-set hold-out approach.
Test-set approach
Here we randomly divide the available set of observations into two parts: a training set and a test or hold-out set (20% of the entire dataset).  
In scikit-learn a random split into training and test sets can be quickly computed with the train_test_split helper function.
End of explanation
model.fit(x_train, y_train)
Explanation: Measure the accuracy of the model
We fit the model on the train dataset:
End of explanation
print('Test Accuracy: {:.3f}'.format(model.score(x_test, y_test)))
Explanation: And measure the accuracy on the test dataset.
Note: the score() function is a shortcut to run predict() and then calculate the accuracy.
End of explanation
from sklearn.model_selection import KFold
import numpy as np
Explanation: The validation estimate of the test error can be highly variable, depending on precisely which observations are included in the training set and which observations are included in the test set. Every time you split the data into different train/test datasets you get different models and scores. You can try it yourself by removing the fixed random seed above.
Cross-validation approach
A solution to this problem is a procedure called cross-validation (CV for short).  
K-Fold
In the basic approach, called K-fold CV, the training set is split into K smaller sets.
We leave out part k, fit the model to the other K-1 parts (combined), and then obtain predictions for the left-out k-th part.
This is done in turn for each part k = 1,2,...K, and then the results are combined.
End of explanation
kf = KFold(n_splits = 10).split(X, Y)
Explanation: KFold divides all the samples in K groups of samples, called folds, of equal sizes (if possible).
End of explanation
scores = []
for k, (train_index, test_index) in enumerate(kf):
    X_train, X_test = X.iloc[train_index], X.iloc[test_index]
    Y_train, Y_test = Y.iloc[train_index], Y.iloc[test_index]
    model.fit(X_train, Y_train)
    scoreK = model.score(X_train, Y_train)
    scores.append(scoreK)
    print('Fold: {:2d}, Acc: {:.3f}'.format(k+1, scoreK))
print('\nCV accuracy: {:.3f} +/- {:.3f}'.format(np.mean(scores), 
                                                np.std(scores)))
Explanation: Note: I use .iloc to index the Pandas dataframe to avoid that some data become NaN after the split and get this error:
ValueError: Input contains NaN, infinity or a value too large for dtype('float64').
End of explanation
from sklearn.model_selection import LeaveOneOut 
LOO = LeaveOneOut().split(X, Y)
Explanation: Leave One Out (LOO) cross-validation
LOO is a special case of the Kfold splitting approach: if K = n, this is equivalent to the Leave One Out strategy.
End of explanation
scores = []
for k, (train_index, test_index) in enumerate(LOO):
    X_train, X_test = X.iloc[train_index], X.iloc[test_index]
    Y_train, Y_test = Y.iloc[train_index], Y.iloc[test_index]
    
    model.fit(X_train, Y_train)
    scoreK = model.score(X_train, Y_train)
    scores.append(scoreK)
print('\nCV accuracy: {:.3f} +/- {:.3f}'.format(np.mean(scores), 
                                                np.std(scores)))
Explanation: Note: not printing each fold here because LOO is computationally more expensive.
End of explanation
from sklearn.model_selection import cross_val_score
# We are passing the entirety of X and y, not X_train or y_train
scores = cross_val_score(model, X, Y, cv=KFold(n_splits=5, shuffle=True,
                                              random_state=1))
print ("Cross-validated scores:", scores)
print('CV accuracy: {:.3f}  +/- {:.3f} '.format (np.mean(scores), np.std(scores)))
Explanation: Now the accuracy of 0.606 is a much more reliable measure than the one from the test-set approach, which could change depending on which test set has been randomly selected.
A generic cross-validation score evaluating function.
The sklearn helper cross_val_score can quickly estimate the score of a model, using one CV approach, therefore substituting the above code with a more compact form:
End of explanation
from sklearn.preprocessing import PolynomialFeatures
poly = PolynomialFeatures(degree=2) # quadratic function
  # we first transform our data
x_train2 = poly.fit_transform(x_train)
x_test2  = poly.fit_transform(x_test)
x_train2[0:5]
Explanation: In the first iteration, the accuracy is 58%, in the second iteration the accuracy is 65% and so on.  
Generally a 10-folds is a good compromise between speed and accuracy.
A model with polynomial features
Now we try to improve our simple model's accuracy by adding polynomial terms to our feature, i.e. transforming it into its power.
End of explanation
model2 = linear_model.LinearRegression()
model2.fit(x_train2, y_train)
model2.score(x_test2, y_test)
Explanation: Then we fit a new model to the transformed data:
End of explanation
poly = PolynomialFeatures(degree=3) # cubic function
x_train3 = poly.fit_transform(x_train) # again, transform the data
x_test3  = poly.fit_transform(x_test)
model3 = linear_model.LinearRegression()
model3.fit(x_train3, y_train) # fit a new model
model3.score(x_test3, y_test)
Explanation: It looks better, so we can try with a cubic function:
End of explanation
# search for an optimal value of poly n for LR
  # range of n we want to try
n_range = range(1, 15)
  # empty list to store scores
n_scores = []
model_n = linear_model.LinearRegression()
  # 1. we will loop through values of n
for n in n_range:
      # 2. transform features with n degree
    poly = PolynomialFeatures(degree=n)
    x_val_n = poly.fit_transform(X)    
      # 3. obtain cross_val_score for model_n
    scores = cross_val_score(model_n, x_val_n, Y, cv=KFold(n_splits=5, 
                                                           shuffle=True,
                                                           random_state=1)) 
    # 4. append mean of scores to n_scores list
    n_scores.append(scores.mean())
    #print ("Cross-validated scores:", scores)
print('CV accuracy: {:.3f} +/- {:.3f} '.format(np.mean(n_scores), np.std(n_scores)))
# in essence, this is basically running the k-fold cross-validation method n times :
print('Length of list', len(n_scores))
print('Max of list', max(n_scores))
# plot how accuracy changes as we vary the polynomial degree
plt.plot(n_range, n_scores)
plt.xlabel('Degree for horsepower')
plt.ylabel('Cross-validated accuracy')
Explanation: These results show that a model which predicts mpg using a quadratic function of horsepower performs better than a model that involves only a linear function of horsepower, and there is little evidence in favour of a model that uses a cubic function of horsepower.
Using the CV approach to tune parameters (model selection)
We can repeat this procedure for increasingly complex polynomial fits. 
This is called model selection.  
To automate the process, we use a for()  loop, which iteratively fits polynomial regressions for polynomials of order i = 1 to i = 15, computes the associated cross-validation score and stores it.
End of explanation
<END_TASK>