<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 Regularized Regression
Step1:  0) Dependencies
Step2:  1) Import dataset
Step3:  1.1) Define KPI column and feature set
Step4:  2) Build RBA Model
Create a linear model to measure the impact of digital media (x variables) on conversions (y variable).
Different regularization techniques, such as Ridge or Lasso, can be implemented to adjust for highly correlated features.
Step5:  2.1) Print the model coefficient results
Step6:  2.2) Print the model evaluation metrics
Step7:  3) Calculate contribution of each digitl
Use the relativeImp package
to conduct key driver analysis and generate relative importance values by feature in the model.
The relativeImp function produces a raw relative importance and a normalized relative importance value. 
- Raw relative importance sums to the r-squared of the linear model.
- Normalized relative importance is scaled to sum to 1
Step8:  4) Validate Linear Regression Model Assumptions
4.1) Generate model residuals
Step9:  4.2) Linearity
Step10:  4.3) Normality of Errors
Step11:  4.4) Absence of Multicollinearity
Tested and checked during data processing stage
4.5) Homoscedasticity
Step12:  4.6) Residual Autocorrelation Check
<ASSISTANT_TASK:>
Python Code:
###########################################################################
#
#  Copyright 2021 Google Inc.
#
#  Licensed under the Apache License, Version 2.0 (the "License");
#  you may not use this file except in compliance with the License.
#  You may obtain a copy of the License at
#
#      https://www.apache.org/licenses/LICENSE-2.0
#
#  Unless required by applicable law or agreed to in writing, software
#  distributed under the License is distributed on an "AS IS" BASIS,
#  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
#  See the License for the specific language governing permissions and
#  limitations under the License.
#
# This solution, including any related sample code or data, is made available 
# on an “as is,” “as available,” and “with all faults” basis, solely for 
# illustrative purposes, and without warranty or representation of any kind. 
# This solution is experimental, unsupported and provided solely for your 
# convenience. Your use of it is subject to your agreements with Google, as 
# applicable, and may constitute a beta feature as defined under those 
# agreements.  To the extent that you make any data available to Google in 
# connection with your use of the solution, you represent and warrant that you 
# have all necessary and appropriate rights, consents and permissions to permit 
# Google to use and process that data.  By using any portion of this solution, 
# you acknowledge, assume and accept all risks, known and unknown, associated 
# with its usage, including with respect to your deployment of any portion of 
# this solution in your systems, or usage in connection with your business, 
# if at all.
###########################################################################
Explanation: Regularized Regression
End of explanation
################################################################################
######################### CHANGE BQ PROJECT NAME BELOW #########################
################################################################################
project_name = '' #add proj name and dataset
# Google credentials authentication libraries
from google.colab import auth
auth.authenticate_user()
# data processing libraries
import numpy as np
from numpy.core.numeric import NaN
import pandas as pd
import pandas_gbq
import datetime
# modeling and metrics
from scipy.optimize import least_squares
from statsmodels.tools.tools import add_constant
import statsmodels.api as sm
from sklearn.model_selection import train_test_split, cross_val_score, LeaveOneOut, KFold, LeavePOut
from sklearn.linear_model import LinearRegression, Ridge, Lasso, ElasticNet
from sklearn.metrics import mean_squared_error, r2_score, mean_absolute_error
from statsmodels.stats.stattools import durbin_watson
# Visualization
import matplotlib.pyplot as plt
%matplotlib inline
import seaborn as sns
# Calculating Relative Importance
!pip install relativeImp
from relativeImp import relativeImp
# BigQuery Magics
'''
BigQuery magics are used to run BigQuery SQL queries in a python environment.
These queries can also be run in the BigQuery UI
'''
from google.cloud import bigquery
from google.cloud.bigquery import magics
magics.context.project = project_name  #update your project name 
client = bigquery.Client(project=magics.context.project)
%load_ext google.cloud.bigquery
bigquery.USE_LEGACY_SQL = False
Explanation: 0) Dependencies
End of explanation
################################################################################
######################### CHANGE BQ PROJECT NAME BELOW #########################
################################################################################
%%bigquery df
SELECT *
FROM `.RBA_demo.cleaned_data`; #update with project name. 
df.head()
Explanation: 1) Import dataset
End of explanation
KPI_COL = "y1" 
y = df[KPI_COL]
X = df[df.columns[df.columns != KPI_COL]].values
Explanation: 1.1) Define KPI column and feature set
End of explanation
reg = Ridge().fit(X,y)
#reg = Lasso().fit(X,y)
#reg = ElasticNet().fit(X,y)
#reg = LinearRegression().fit(X,y)
Explanation: 2) Build RBA Model
Create a linear model to measure the impact of digital media (x variables) on conversions (y variable).
Different regularization techniques, such as Ridge or Lasso, can be implemented to adjust for highly correlated features.
End of explanation
reg.intercept_
coefficients = reg.coef_.tolist()
coefficients
Explanation: 2.1) Print the model coefficient results
End of explanation
#R-squared
reg.score(X,y)
# Generate predictions to calculate MAE, MSE, RMSE
Y_prediction = reg.predict(X)
mean_absolute_error(y,Y_prediction)
mean_squared_error(y,Y_prediction)
rmse = np.sqrt(mean_squared_error(y,Y_prediction))
rmse
Explanation: 2.2) Print the model evaluation metrics
End of explanation
yName = 'y1'
xNames = df[df.columns[df.columns != KPI_COL]].columns.to_list()
df_results = relativeImp(df, outcomeName = yName, driverNames = xNames)
df_results
Explanation: 3) Calculate contribution of each digitl
Use the relativeImp package
to conduct key driver analysis and generate relative importance values by feature in the model.
The relativeImp function produces a raw relative importance and a normalized relative importance value. 
- Raw relative importance sums to the r-squared of the linear model.
- Normalized relative importance is scaled to sum to 1
End of explanation
residuals = Y_prediction - y
Explanation: 4) Validate Linear Regression Model Assumptions
4.1) Generate model residuals
End of explanation
'''
Visually inspect linearity between target variable (y1) and predictions
'''
plt.plot(Y_prediction,y,'o',alpha=0.5)
plt.show()
Explanation: 4.2) Linearity
End of explanation
'''
Visually inspect the residuals to confirm normality
'''
fig = sm.qqplot(residuals)
sns.kdeplot(residuals, label = '', shade = True)
plt.xlabel('Model Residuals'); plt.ylabel('Density'); plt.title('Distribution of Residuals');
Explanation: 4.3) Normality of Errors
End of explanation
'''
Visually inspect residuals to confirm constant variance
'''
plt.plot(residuals,'o',alpha=0.5)
plt.show()
Explanation: 4.4) Absence of Multicollinearity
Tested and checked during data processing stage
4.5) Homoscedasticity
End of explanation
'''
The Durbin Watson test is a statistical test for detecting autocorrelation of the 
model residuals
'''
dw = durbin_watson(residuals)
print('Durbin-Watson',dw)
if dw < 1.5:
        print('Positive autocorrelation', '\n')
elif dw > 2.5:
        print('Negative autocorrelation', '\n')
else:
        print('Little to no autocorrelation', '\n')
Explanation: 4.6) Residual Autocorrelation Check
End of explanation
<END_TASK>