<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 Supervised Learning Part 2 -- Regression Analysis
In regression we are trying to predict a continuous output variable -- in contrast to the nominal variables we were predicting in the previous classification examples. 
Let's start with a simple toy example with one feature dimension (explanatory variable) and one target variable. We will create a dataset out of a sine curve with some noise
Step1:  Linear Regression
The first model that we will introduce is the so-called simple linear regression. Here, we want to fit a line to the data, which 
One of the simplest models again is a linear one, that simply tries to predict the data as lying on a line. One way to find such a line is LinearRegression (also known as Ordinary Least Squares (OLS) regression).
The interface for LinearRegression is exactly the same as for the classifiers before, only that y now contains float values, instead of classes.
As we remember, the scikit-learn API requires us to provide the target variable (y) as a 1-dimensional array; scikit-learn's API expects the samples (X) in form a 2-dimensional array -- even though it may only consist of 1 feature. Thus, let us convert the 1-dimensional x NumPy array into an X array with 2 axes
Step2:  Again, we start by splitting our dataset into a training (75%) and a test set (25%)
Step3:  Next, we use the learning algorithm implemented in LinearRegression to fit a regression model to the training data
Step4:  After fitting to the training data, we paramerterized a linear regression model with the following values.
Step5:  Since our regression model is a linear one, the relationship between the target variable (y) and the feature variable (x) is defined as 
$$y = weight \times x + \text{intercept}$$.
Plugging in the min and max values into thos equation, we can plot the regression fit to our training data
Step6:  Similar to the estimators for classification in the previous notebook, we use the predict method to predict the target variable. And we expect these predicted values to fall onto the line that we plotted previously
Step7:  As we can see in the plot above, the line is able to capture the general slope of the data, but not many details.
Next, let's try the test set
Step8:  Again, scikit-learn provides an easy way to evaluate the prediction quantitatively using the score method. For regression tasks, this is the R<sup>2</sup> score. Another popular way would be the Mean Squared Error (MSE). As its name implies, the MSE is simply the average squared difference over the predicted and actual target values
$$MSE = \frac{1}{n} \sum_{i=1}^{n} (\text{predicted}_i - \text{true}_i)^2$$
Step9:  <div class="alert alert-success">
    <b>EXERCISE</b>
Step10:  KNeighborsRegression
As for classification, we can also use a neighbor based method for regression. We can simply take the output of the nearest point, or we could average several nearest points. This method is less popular for regression than for classification, but still a good baseline.
Step11:  Again, let us look at the behavior on training and test set
Step12:  On the training set, we do a perfect job
Step13:  On the test set, we also do a better job of capturing the variation, but our estimates look much messier than before.
Let us look at the R<sup>2</sup> score
Step14:  Much better than before! Here, the linear model was not a good fit for our problem; it was lacking in complexity and thus under-fit our data.
<div class="alert alert-success">
    <b>EXERCISE</b>
<ASSISTANT_TASK:>
Python Code:
x = np.linspace(-3, 3, 100)
print(x)
rng = np.random.RandomState(42)
y = np.sin(4 * x) + x + rng.uniform(size=len(x))
plt.plot(x, y, 'o');
Explanation: Supervised Learning Part 2 -- Regression Analysis
In regression we are trying to predict a continuous output variable -- in contrast to the nominal variables we were predicting in the previous classification examples. 
Let's start with a simple toy example with one feature dimension (explanatory variable) and one target variable. We will create a dataset out of a sine curve with some noise:
End of explanation
print('Before: ', x.shape)
X = x[:, np.newaxis]
print('After: ', X.shape)
Explanation: Linear Regression
The first model that we will introduce is the so-called simple linear regression. Here, we want to fit a line to the data, which 
One of the simplest models again is a linear one, that simply tries to predict the data as lying on a line. One way to find such a line is LinearRegression (also known as Ordinary Least Squares (OLS) regression).
The interface for LinearRegression is exactly the same as for the classifiers before, only that y now contains float values, instead of classes.
As we remember, the scikit-learn API requires us to provide the target variable (y) as a 1-dimensional array; scikit-learn's API expects the samples (X) in form a 2-dimensional array -- even though it may only consist of 1 feature. Thus, let us convert the 1-dimensional x NumPy array into an X array with 2 axes:
End of explanation
from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, random_state=42)
Explanation: Again, we start by splitting our dataset into a training (75%) and a test set (25%):
End of explanation
from sklearn.linear_model import LinearRegression
regressor = LinearRegression()
regressor.fit(X_train, y_train)
Explanation: Next, we use the learning algorithm implemented in LinearRegression to fit a regression model to the training data:
End of explanation
print('Weight coefficients: ', regressor.coef_)
print('y-axis intercept: ', regressor.intercept_)
Explanation: After fitting to the training data, we paramerterized a linear regression model with the following values.
End of explanation
min_pt = X.min() * regressor.coef_[0] + regressor.intercept_
max_pt = X.max() * regressor.coef_[0] + regressor.intercept_
plt.plot([X.min(), X.max()], [min_pt, max_pt])
plt.plot(X_train, y_train, 'o');
Explanation: Since our regression model is a linear one, the relationship between the target variable (y) and the feature variable (x) is defined as 
$$y = weight \times x + \text{intercept}$$.
Plugging in the min and max values into thos equation, we can plot the regression fit to our training data:
End of explanation
y_pred_train = regressor.predict(X_train)
plt.plot(X_train, y_train, 'o', label="data")
plt.plot(X_train, y_pred_train, 'o', label="prediction")
plt.plot([X.min(), X.max()], [min_pt, max_pt], label='fit')
plt.legend(loc='best')
Explanation: Similar to the estimators for classification in the previous notebook, we use the predict method to predict the target variable. And we expect these predicted values to fall onto the line that we plotted previously:
End of explanation
y_pred_test = regressor.predict(X_test)
plt.plot(X_test, y_test, 'o', label="data")
plt.plot(X_test, y_pred_test, 'o', label="prediction")
plt.plot([X.min(), X.max()], [min_pt, max_pt], label='fit')
plt.legend(loc='best');
Explanation: As we can see in the plot above, the line is able to capture the general slope of the data, but not many details.
Next, let's try the test set:
End of explanation
regressor.score(X_test, y_test)
Explanation: Again, scikit-learn provides an easy way to evaluate the prediction quantitatively using the score method. For regression tasks, this is the R<sup>2</sup> score. Another popular way would be the Mean Squared Error (MSE). As its name implies, the MSE is simply the average squared difference over the predicted and actual target values
$$MSE = \frac{1}{n} \sum_{i=1}^{n} (\text{predicted}_i - \text{true}_i)^2$$
End of explanation
# %load solutions/06B_lin_with_sine.py
Explanation: <div class="alert alert-success">
    <b>EXERCISE</b>:
     <ul>
      <li>
      Add a feature containing  `sin(4x)` to `X` and redo the fit. Visualize the predictions with this new richer, yet linear, model.
      </li>
    </ul>
</div>
End of explanation
from sklearn.neighbors import KNeighborsRegressor
kneighbor_regression = KNeighborsRegressor(n_neighbors=1)
kneighbor_regression.fit(X_train, y_train)
Explanation: KNeighborsRegression
As for classification, we can also use a neighbor based method for regression. We can simply take the output of the nearest point, or we could average several nearest points. This method is less popular for regression than for classification, but still a good baseline.
End of explanation
y_pred_train = kneighbor_regression.predict(X_train)
plt.plot(X_train, y_train, 'o', label="data", markersize=10)
plt.plot(X_train, y_pred_train, 's', label="prediction", markersize=4)
plt.legend(loc='best');
Explanation: Again, let us look at the behavior on training and test set:
End of explanation
y_pred_test = kneighbor_regression.predict(X_test)
plt.plot(X_test, y_test, 'o', label="data", markersize=8)
plt.plot(X_test, y_pred_test, 's', label="prediction", markersize=4)
plt.legend(loc='best');
Explanation: On the training set, we do a perfect job: each point is its own nearest neighbor!
End of explanation
kneighbor_regression.score(X_test, y_test)
Explanation: On the test set, we also do a better job of capturing the variation, but our estimates look much messier than before.
Let us look at the R<sup>2</sup> score:
End of explanation
# %load solutions/06A_knn_vs_linreg.py
Explanation: Much better than before! Here, the linear model was not a good fit for our problem; it was lacking in complexity and thus under-fit our data.
<div class="alert alert-success">
    <b>EXERCISE</b>:
     <ul>
      <li>
      Compare the KNeighborsRegressor and LinearRegression on the boston housing dataset. You can load the dataset using ``sklearn.datasets.load_boston``. You can learn about the dataset by reading the ``DESCR`` attribute.
      </li>
    </ul>
</div>
End of explanation
<END_TASK>