<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 Initialise the libraries
Step1:  Load the data
Step2:  Feature engineering
Step3:  LASSO
Step4:  Calculate the RSS for best L1 on test data
Step5:  2 Phase LASSO for finding a desired number of nonzero features
First we loop through an array of L1 alues and find an interval where our model gets the desired number of nonzero coefficients. In phase 2 we loop once more through the interval (from the first phase) but with small steps, the goal of the second phase is to find the optimal L1 for our nonzero number.
<ASSISTANT_TASK:>
Python Code:
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
from math import log, sqrt
from sklearn import linear_model
Explanation: Initialise the libraries
End of explanation
dtype_dict = {'bathrooms':float, 'waterfront':int, 'sqft_above':int, 'sqft_living15':float, 'grade':int, 'yr_renovated':int, 'price':float, 'bedrooms':float, 'zipcode':str, 'long':float, 'sqft_lot15':float, 'sqft_living':float, 'floors':float, 'condition':int, 'lat':float, 'date':str, 'sqft_basement':int, 'yr_built':int, 'id':str, 'sqft_lot':int, 'view':int}
sales = pd.read_csv('../datasets/kc_house_data.csv', dtype=dtype_dict)
testing = pd.read_csv('../datasets/wk3_kc_house_test_data.csv', dtype=dtype_dict)
training = pd.read_csv('../datasets/wk3_kc_house_train_data.csv', dtype=dtype_dict)
validation = pd.read_csv('../datasets/wk3_kc_house_valid_data.csv', dtype=dtype_dict)
Explanation: Load the data
End of explanation
sales['sqft_living_sqrt'] = sales['sqft_living'].apply(sqrt)
sales['sqft_lot_sqrt'] = sales['sqft_lot'].apply(sqrt)
sales['bedrooms_square'] = sales['bedrooms']*sales['bedrooms']
sales['floors_square'] = sales['floors']*sales['floors']
testing['sqft_living_sqrt'] = testing['sqft_living'].apply(sqrt)
testing['sqft_lot_sqrt'] = testing['sqft_lot'].apply(sqrt)
testing['bedrooms_square'] = testing['bedrooms']*testing['bedrooms']
testing['floors_square'] = testing['floors']*testing['floors']
training['sqft_living_sqrt'] = training['sqft_living'].apply(sqrt)
training['sqft_lot_sqrt'] = training['sqft_lot'].apply(sqrt)
training['bedrooms_square'] = training['bedrooms']*training['bedrooms']
training['floors_square'] = training['floors']*training['floors']
validation['sqft_living_sqrt'] = validation['sqft_living'].apply(sqrt)
validation['sqft_lot_sqrt'] = validation['sqft_lot'].apply(sqrt)
validation['bedrooms_square'] = validation['bedrooms']*validation['bedrooms']
validation['floors_square'] = validation['floors']*validation['floors']
all_features = ['bedrooms', 'bedrooms_square',
            'bathrooms',
            'sqft_living', 'sqft_living_sqrt',
            'sqft_lot', 'sqft_lot_sqrt',
            'floors', 'floors_square',
            'waterfront', 'view', 'condition', 'grade',
            'sqft_above',
            'sqft_basement',
            'yr_built', 'yr_renovated']
Explanation: Feature engineering
End of explanation
model_all = linear_model.Lasso(alpha=5e2, normalize=True) # set parameters
model_all.fit(sales[all_features], sales['price']) # learn weights
print (all_features, '\n')
print (model_all.coef_)
print ('With intercetp, nonzeros: ', np.count_nonzero(model_all.coef_) + np.count_nonzero(model_all.intercept_))
print ('No intercept: ', np.count_nonzero(model_all.coef_))
l1pentalities = np.logspace(1, 7, num=13)
l1pentalities
import sys
minRSS = sys.maxsize
bestL1 = 0
for l1p in l1pentalities:
    model = linear_model.Lasso(alpha=l1p, normalize=True)
    model.fit(training[all_features], training['price']) # learn weights
    
    # Calculate the RSS
    RSS = ((model.predict(validation[all_features]) - validation.price) ** 2).sum()
    print ("RSS: ", RSS, " for L1: ", l1p)
    
    # Remember the min RSS
    if RSS < minRSS:
        minRSS = RSS
        bestL1 = l1p
        
print ('\nMinimum RSS:', minRSS, ' for L1: ', bestL1)
Explanation: LASSO
End of explanation
model = linear_model.Lasso(alpha=bestL1, normalize=True)
model.fit(training[all_features], training['price']) # learn weights
# Calculate the RSS
RSS = ((model.predict(testing[all_features]) - testing.price) ** 2).sum()
print ("RSS: ", RSS, " for L1: ", bestL1)
print (model.coef_)
print (model.intercept_)
np.count_nonzero(model.coef_) + np.count_nonzero(model.intercept_)
Explanation: Calculate the RSS for best L1 on test data
End of explanation
max_nonzeros = 7
L1_range = np.logspace(1, 4, num=20)
print (L1_range)
l1_penalty_min = L1_range[0]
l1_penalty_max = L1_range[19]
for L1 in L1_range:
    model = linear_model.Lasso(alpha=L1, normalize=True)
    model.fit(training[all_features], training['price']) # learn weights
    
    nonZeroes = np.count_nonzero(model.coef_) + np.count_nonzero(model.intercept_)
    
    # The largest l1_penalty that has more non-zeros than ‘max_nonzeros’
    if (nonZeroes > max_nonzeros) and (L1 > l1_penalty_min) :
        l1_penalty_min = L1
    
    # The smallest l1_penalty that has fewer non-zeros than ‘max_nonzeros’
    if (nonZeroes < max_nonzeros) and (L1 < l1_penalty_max):
        l1_penalty_max = L1
    
print ('l1_penalty_min: ', l1_penalty_min)
print ('l1_penalty_max: ', l1_penalty_max)
L1_narrow_interval = np.linspace(l1_penalty_min,l1_penalty_max,20)
print (L1_narrow_interval)
minRSS = sys.maxsize
bestL1 = 0
# Find the model with desired number of nonzeroes with the best RSS
for L1 in L1_narrow_interval:
    model = linear_model.Lasso(alpha=L1, normalize=True)
    model.fit(validation[all_features], validation['price']) # learn weights
    
    # Calculate the RSS
    RSS = ((model.predict(validation[all_features]) - validation.price) ** 2).sum()
    print ("RSS: ", RSS, " for L1: ", L1)
     
    # Remember the min RSS
    if RSS < minRSS:
        minRSS = RSS
        bestL1 = L1
        
print ('\nMinimum RSS:', minRSS, ' for L1: ', bestL1)
model = linear_model.Lasso(alpha=bestL1, normalize=True)
model.fit(validation[all_features], validation['price']) # learn weights
print (all_features)
print ()
print (model.coef_)
print ()
print (model.intercept_)
Explanation: 2 Phase LASSO for finding a desired number of nonzero features
First we loop through an array of L1 alues and find an interval where our model gets the desired number of nonzero coefficients. In phase 2 we loop once more through the interval (from the first phase) but with small steps, the goal of the second phase is to find the optimal L1 for our nonzero number.
End of explanation
<END_TASK>