<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 The overfitting problem
We have seen what is linear regression, how to make models and algorithms for estimating the parameters of such models, how to measure the loss.
Now we see how to assess how well the considered method should perform in predicting new data, how to select amongst possible models to choose the best performing.
This leads directly to the bias-variance tradeoff, which is fundamental to machine learning. 
Load in the data
The dataset is from house sales in King County, the region where the city of Seattle, WA is located.
Step1:  The dataset contains information (21 features, including the price) related to 21613 houses.
Our target variable (i.e., what we want to predict when a new house gets on sale) is the price.
Baseline
Step2:  The predictions are very easy to calculate, just the baseline value
Step3:  Example
Step5:  The estimated price for the example house will still be around 540K, wile the real value is around 222K. Quite an error!
Measures of loss
There are several way of implementing the loss, I use the squared error here.  
$L = [y - f(X)]^2$
Step6:  To better see the value of the cost function we use also the RMSE, the Root Mean Square Deviation.
Basically the average of the losses, rooted.
Step7:  As you can see, it is quite high error, especially related to the average selling price.
Now, we can look at how training error behaves as model complexity increases.
Learning a better but still simple model
Using a constant value, the average, is easy but does not make too much sense.
Let's create a linear model with the house size as the feature. We expect that the price is dependent on the size
Step8:  Now that we have fit the model we can extract the regression weights (coefficients) as follows
Step9:  This means that our simple model to predict a house price y is (approximated)
Step10:  We are getting closer to the real value for the example house (recall, it's around 222K).
Compute the Training Error
Now that we can make predictions given the model, let's again compute the RSS and the RMSE.
Step11:  The simple model reduced greatly the training error.
Learning a multiple regression model
We can add more features to the model, for example the number of bedrooms and bathrooms.
Step12:  We can learn a multiple regression model predicting 'price' based on the above features on the data with the following code
Step13:  Now that we have fitted the model we can extract the regression weights (coefficients) as follows
Step14:  The better model is therefore
Step15:  Again, a little bit closer to the real value (222K)
Compute the Training Error
Now that we can make predictions given the model, let's write a function to compute the RSS of the model.
Step16:  Only a slight improvement this time
Create some new features
Although we often think of multiple regression as including multiple different features (e.g. # of bedrooms, squarefeet, and # of bathrooms) but we can also consider transformations of existing features e.g. the log of the squarefeet or even "interaction" features such as the product of bedrooms and bathrooms.
Step17:  Next we create the following new features as column 
Step18:  Squaring bedrooms will increase the separation between not many bedrooms (e.g. 1) and lots of bedrooms (e.g. 4) since 1^2 = 1 but 4^2 = 16. Consequently this feature will mostly affect houses with many bedrooms.
bedrooms times bathrooms gives what's called an "interaction" feature. It is large when both of them are large.
Taking the log of squarefeet has the effect of bringing large values closer together and spreading out small values.
Adding latitude to longitude is totally non-sensical but we will do it anyway (you'll see why)
Learning Multiple Models
Now we will learn the weights for five (nested) models for predicting house prices. The first model will have the fewest features, the second model will add more features and so on
Step19:  Now that we have the features, we learn the weights for the five different models for predicting target = 'price' using and look at the value of the weights/coefficients
Step20:  Interesting
Step21:  model_5 has the lowest RSS on the training data.
The most complex model.
The test error
Training error decreases quite significantly with model complexity. This is quite intuitive, because the model was fit on the training points and then as we increase the model complexity, we are better able to fit the training data points.
A natural question is whether a training error is a good measure of predictive performance?  
The issue is that the training error is overly optimistic and that's because the beta parameters were fit on the training data to minimise the residual sum of squares, which can often be related to the training error.
So, in general, having small training error does not imply having good predictive performance.
This takes us to something called test error (or out-of-sample error)
Step22:  In this case the testing set will be the 30% (therefore the training set is 70% of the original data)
Step23:  Retrain the models on training data only
Step24:  Now compute the RSS on TEST data for each of the models.
<ASSISTANT_TASK:>
Python Code:
import pandas as pd
dtype_dict = {'bathrooms':float, 'waterfront':int, 'sqft_above':int, 'sqft_living15':float, 
              'grade':int, 'yr_renovated':int, 'price':float, 'bedrooms':float, 'zipcode':str, 
              'long':float, 'sqft_lot15':float, 'sqft_living':float, 'floors':str, 
              'condition':int, 'lat':float, 'date':str, 'sqft_basement':int, 'yr_built':int, 
              'id':str, 'sqft_lot':int, 'view':int}
sales = pd.read_csv('../datasets/kc_house_data.csv', dtype=dtype_dict)
sales.head()
sales.shape
m = sales.shape[0] # number of training examples
m
Explanation: The overfitting problem
We have seen what is linear regression, how to make models and algorithms for estimating the parameters of such models, how to measure the loss.
Now we see how to assess how well the considered method should perform in predicting new data, how to select amongst possible models to choose the best performing.
This leads directly to the bias-variance tradeoff, which is fundamental to machine learning. 
Load in the data
The dataset is from house sales in King County, the region where the city of Seattle, WA is located.
End of explanation
# Let's compute the mean of the House Prices in King County 
y = sales['price'] # extract the price column
avg_price = y.mean() # this is our baseline
print ("average price: ${:.0f} ".format(avg_price))
ExamplePrice = y[0]
ExamplePrice
Explanation: The dataset contains information (21 features, including the price) related to 21613 houses.
Our target variable (i.e., what we want to predict when a new house gets on sale) is the price.
Baseline: the simplest model
Now let's compute the loss in the case of the simplest model: a fixed price equal to the average of historic prices, independently on house size, rooms, location, ...
End of explanation
def get_baseline_predictions():
       # Simplest version: return the baseline as predicted values
    predicted_values = avg_price 
    return predicted_values
Explanation: The predictions are very easy to calculate, just the baseline value:
End of explanation
my_house_size = 2500
estimated_price = get_baseline_predictions()
print ("The estimated price for a house with {} squared feet is {:.0f}".format(my_house_size, estimated_price))
Explanation: Example:
End of explanation
import numpy as np
def get_loss(yhat, target):
    
    Arguments:
    yhat -- vector of size m (predicted labels)
    target -- vector of size m (true labels)
    
    Returns:
    loss -- the value of the L2 loss function
    
      # compute the residuals (since we are squaring it doesn't matter 
      #   which order you subtract)
      # np.dot will square the residuals and add them up
    loss = np.dot((target - yhat), (target - yhat))
    return(loss)
Explanation: The estimated price for the example house will still be around 540K, wile the real value is around 222K. Quite an error!
Measures of loss
There are several way of implementing the loss, I use the squared error here.  
$L = [y - f(X)]^2$
End of explanation
baselineCost = get_loss(get_baseline_predictions(), y)
print ("Training Error for baseline RSS: {:.0f}".format(baselineCost))
print ("Average Training Error for baseline RMSE: {:.0f}".format(np.sqrt(baselineCost/m)))
Explanation: To better see the value of the cost function we use also the RMSE, the Root Mean Square Deviation.
Basically the average of the losses, rooted.
End of explanation
from sklearn import linear_model
simple_model = linear_model.LinearRegression()
simple_features = sales[['sqft_living']] # input X: the house size
simple_model.fit(simple_features, y)
Explanation: As you can see, it is quite high error, especially related to the average selling price.
Now, we can look at how training error behaves as model complexity increases.
Learning a better but still simple model
Using a constant value, the average, is easy but does not make too much sense.
Let's create a linear model with the house size as the feature. We expect that the price is dependent on the size: bigger house, more expensive.
End of explanation
simple_model_intercept = simple_model.intercept_
print (simple_model_intercept)
simple_model_weights = simple_model.coef_
print (simple_model_weights)
Explanation: Now that we have fit the model we can extract the regression weights (coefficients) as follows:
End of explanation
training_predictions = simple_model.predict(simple_features)
print (training_predictions[0])
Explanation: This means that our simple model to predict a house price y is (approximated):  
$y = -43581 + 281x $
where x is the size in squared feet.
It is not anymore a horizontal line but a diagonal one, with a slope.
Making Predictions
Recall that once a model is built we can use the .predict() function to find the predicted values for data we pass. For example using the example model above:
End of explanation
    # First get the predictions using the features subset
predictions = simple_model.predict(sales[['sqft_living']])
simpleCost = get_loss(predictions, y) 
print ("Training Error for baseline RSS: {:.0f}".format(simpleCost))
print ("Average Training Error for baseline RMSE: {:.0f}".format(np.sqrt(simpleCost/m)))
Explanation: We are getting closer to the real value for the example house (recall, it's around 222K).
Compute the Training Error
Now that we can make predictions given the model, let's again compute the RSS and the RMSE.
End of explanation
more_features = sales[['sqft_living', 'bedrooms', 'bathrooms']] # input X
Explanation: The simple model reduced greatly the training error.
Learning a multiple regression model
We can add more features to the model, for example the number of bedrooms and bathrooms.
End of explanation
better_model = linear_model.LinearRegression()
better_model.fit(more_features, y)
Explanation: We can learn a multiple regression model predicting 'price' based on the above features on the data with the following code:
End of explanation
betterModel_intercept = better_model.intercept_
print (betterModel_intercept)
betterModel_weights = better_model.coef_
print (betterModel_weights)
Explanation: Now that we have fitted the model we can extract the regression weights (coefficients) as follows:
End of explanation
better_predictions = better_model.predict(more_features)
print (better_predictions[0]) 
Explanation: The better model is therefore:
$y = 74847 + 309x1 - 57861x2 + 7933x3$
Note that the equation has now three variables: the size, the bedrooms and the bathrooms.
Making Predictions
Again we can use the .predict() function to find the predicted values for data we pass. For the model above:
End of explanation
predictions = better_model.predict(more_features)
betterCost = get_loss(predictions, y) 
print ("Training Error for baseline RSS: {:.0f}".format(betterCost))
print ("Average Training Error for baseline RMSE: {:.0f}".format(np.sqrt(betterCost/m)))
Explanation: Again, a little bit closer to the real value (222K)
Compute the Training Error
Now that we can make predictions given the model, let's write a function to compute the RSS of the model.
End of explanation
from math import log
Explanation: Only a slight improvement this time
Create some new features
Although we often think of multiple regression as including multiple different features (e.g. # of bedrooms, squarefeet, and # of bathrooms) but we can also consider transformations of existing features e.g. the log of the squarefeet or even "interaction" features such as the product of bedrooms and bathrooms.
End of explanation
sales['bedrooms_squared'] = sales['bedrooms'].apply(lambda x: x**2)
sales['bed_bath_rooms'] = sales['bedrooms'] * sales.bathrooms
sales['log_sqft_living'] = sales['sqft_living'].apply(lambda x: log(x))
sales['lat_plus_long'] = sales['lat'] + sales.long
sales['bedrooms_4'] = sales['bedrooms'].apply(lambda x: x**4)
sales['bathrooms_7'] = sales['bathrooms'].apply(lambda x: x**7)
sales['size_3'] = sales['sqft_living'].apply(lambda x: x**3)
sales.head()
Explanation: Next we create the following new features as column :
* bedrooms_squared = bedrooms*bedrooms
* bed_bath_rooms = bedrooms*bathrooms
* log_sqft_living = log(sqft_living)
* lat_plus_long = lat + long 
* more polynomial features: bedrooms ^ 4, bathrooms ^ 7, size ^ 3
End of explanation
model_1_features = ['sqft_living', 'bedrooms', 'bathrooms', 'lat', 'long', 'sqft_lot', 'floors']
model_2_features = model_1_features + ['log_sqft_living', 'bedrooms_squared', 'bed_bath_rooms']
model_3_features = model_2_features + ['lat_plus_long']
model_4_features = model_3_features + ['bedrooms_4', 'bathrooms_7']
model_5_features = model_4_features + ['size_3']
Explanation: Squaring bedrooms will increase the separation between not many bedrooms (e.g. 1) and lots of bedrooms (e.g. 4) since 1^2 = 1 but 4^2 = 16. Consequently this feature will mostly affect houses with many bedrooms.
bedrooms times bathrooms gives what's called an "interaction" feature. It is large when both of them are large.
Taking the log of squarefeet has the effect of bringing large values closer together and spreading out small values.
Adding latitude to longitude is totally non-sensical but we will do it anyway (you'll see why)
Learning Multiple Models
Now we will learn the weights for five (nested) models for predicting house prices. The first model will have the fewest features, the second model will add more features and so on:
End of explanation
model_1 = linear_model.LinearRegression()
model_1.fit(sales[model_1_features], y)
model_2 = linear_model.LinearRegression()
model_2.fit(sales[model_2_features], y)
model_3 = linear_model.LinearRegression()
model_3.fit(sales[model_3_features], y)
model_4 = linear_model.LinearRegression()
model_4.fit(sales[model_4_features], y)
model_5 = linear_model.LinearRegression()
model_5.fit(sales[model_5_features], y)
# You can examine/extract each model's coefficients, for example:
print (model_1.coef_)
print (model_2.coef_)
Explanation: Now that we have the features, we learn the weights for the five different models for predicting target = 'price' using and look at the value of the weights/coefficients:
End of explanation
# Compute the RSS for each of the models:
print (get_loss(model_1.predict(sales[model_1_features]), y))
print (get_loss(model_2.predict(sales[model_2_features]), y))
print (get_loss(model_3.predict(sales[model_3_features]), y))
print (get_loss(model_4.predict(sales[model_4_features]), y))
print (get_loss(model_5.predict(sales[model_5_features]), y))
Explanation: Interesting: in the previous model the weight coefficient for the size lot was positive but now in the model_2 is negative.
This is an effect of adding the feature logging the size.
Comparing multiple models
Now that you've learned three models and extracted the model weights we want to evaluate which model is best.
We can use the loss function from earlier to compute the RSS on training data for each of the models.
End of explanation
from sklearn.model_selection import train_test_split
train_data,test_data = train_test_split(sales, test_size=0.3, random_state=999)
train_data.head()
train_data.shape
# test_data = pd.read_csv('kc_house_test_data.csv', dtype=dtype_dict)
test_data.head()
test_data.shape
Explanation: model_5 has the lowest RSS on the training data.
The most complex model.
The test error
Training error decreases quite significantly with model complexity. This is quite intuitive, because the model was fit on the training points and then as we increase the model complexity, we are better able to fit the training data points.
A natural question is whether a training error is a good measure of predictive performance?  
The issue is that the training error is overly optimistic and that's because the beta parameters were fit on the training data to minimise the residual sum of squares, which can often be related to the training error.
So, in general, having small training error does not imply having good predictive performance.
This takes us to something called test error (or out-of-sample error): we hold out some houses from the data set and we're putting these into what's called a test set.
And when we fit our models, we just fit our models on the training data set.
But then when we go to assess our performance of that model we look at these test houses in the test dataset and these are hopefully serving as a proxy of everything out there in the world.
Bottom line, the test error is a (noisy) approximation of the true error.
Split data into training and testing.
Let's see how can be applied to our example.
First we split the data into a training set and a testing set using a function from sklearn, the train_test_split().
We use a seed for reproducibility.
End of explanation
train_y = train_data.price # extract the price column
test_y = test_data.price
Explanation: In this case the testing set will be the 30% (therefore the training set is 70% of the original data)
End of explanation
model_1.fit(train_data[model_1_features], train_y)
model_2.fit(train_data[model_2_features], train_y)
model_3.fit(train_data[model_3_features], train_y)
model_4.fit(train_data[model_4_features], train_y)
model_5.fit(train_data[model_5_features], train_y)
# Compute the RSS on TRAINING data for each of the models
print (get_loss(model_1.predict(train_data[model_1_features]), train_y))
print (get_loss(model_2.predict(train_data[model_2_features]), train_y))
print (get_loss(model_3.predict(train_data[model_3_features]), train_y))
print (get_loss(model_4.predict(train_data[model_4_features]), train_y))
print (get_loss(model_5.predict(train_data[model_5_features]), train_y))
Explanation: Retrain the models on training data only:
End of explanation
# Compute the RSS on TESTING data for each of the three models and record the values:
print (get_loss(model_1.predict(test_data[model_1_features]), test_y))
print (get_loss(model_2.predict(test_data[model_2_features]), test_y))
print (get_loss(model_3.predict(test_data[model_3_features]), test_y))
print (get_loss(model_4.predict(test_data[model_4_features]), test_y))
print (get_loss(model_5.predict(test_data[model_5_features]), test_y))
Explanation: Now compute the RSS on TEST data for each of the models.
End of explanation
<END_TASK>