<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 Linear Regression
Step1:  Cost Function
Step2:  Manual exploration of different values of W and b
Step3:  Linear Regression with Keras
Step4:  Evaluating Model Performance
Step5:  Train Test Split
Step6:  Classification
Step7:  Train/Test split
Step8:  Cross Validation
Step9:  Confusion Matrix
Step10:  Feature Preprocessing
Categorical Features
Step11:  Feature Transformations
1) Rescale with fixed factor
Step12:  MinMax normalization
Step13:  3) Standard normalization
Step14:  Machine Learning Exercises
Exercise 1
You've just been hired at a real estate investment firm and they would like you to build a model for pricing houses. You are given a dataset that contains data for house prices and a few features like number of bedrooms, size in square feet and age of the house. Let's see if you can build a model that is able to predict the price. In this exercise we extend what we have learned about linear regression to a dataset with more than one feature. Here are the steps to complete it
Step15:  MinMaxScaler
Step16:  Lowering the learning rate
Step17:  Using SGD
Step18:  Exercise 2
Your boss was extremely happy with your work on the housing price prediction model and decided to entrust you with a more challenging task. They've seen a lot of people leave the company recently and they would like to understand why that's happening. They have collected historical data on employees and they would like you to build a model that is able to predict which employee will leave next. The would like a model that is better than random guessing. They also prefer false negatives than false positives, in this first phase. Fields in the dataset include
<ASSISTANT_TASK:>
Python Code:
%matplotlib inline
import matplotlib.pyplot as plt
import pandas as pd
import numpy as np
df = pd.read_csv('./data/weight-height.csv')
df.head()
df.plot(kind = 'scatter',
        figsize = (7, 7),
        x = 'Height',
        y = 'Weight',
        title = 'Weight and Height in adults')
df.plot(kind = 'scatter',
        figsize = (7, 7),
        x = 'Height',
        y = 'Weight',
        title = 'Weight and Height in adults')
# Here we're plotting the red line 'by hand' with fixed values
# We'll try to learn this line with an algorithm below
plt.plot([55, 78], [75, 250], color='red', linewidth=3)
def line(x, w=0, b=0):
    return x * w + b
x = np.linspace(55, 80, 100)
x
yhat = line(x, w = 0, b = 0)
yhat
df.plot(kind = 'scatter',
        figsize = (7, 7),
        x = 'Height',
        y = 'Weight',
        title = 'Weight and Height in adults')
plt.plot(x, yhat, color='red', linewidth=3)
Explanation: Linear Regression
End of explanation
def mean_squared_error(y_true, y_pred):
    s = (y_true - y_pred) ** 2
    return s.mean()
X = df[['Height']].values
y_true = df['Weight'].values
y_true
y_pred = line(X)
y_pred
mean_squared_error(y_true, y_pred.ravel())
Explanation: Cost Function
End of explanation
plt.figure(figsize=(10, 5))
# we are going to draw 2 plots in the same figure
# first plot, data and a few lines
ax1 = plt.subplot(121)
df.plot(kind = 'scatter',
        x = 'Height',
        y = 'Weight',
        title = 'Weight and Height in adults', ax=ax1)
# let's explore the cost function for a few values of b between -100 and +150
bbs = np.array([-100, -50, 0, 50, 100, 150])
mses = []  # we will append the values of the cost here, for each line
for b in bbs:
    y_pred = line(X, w = 2, b = b)
    mse = mean_squared_error(y_true, y_pred)
    mses.append(mse)
    plt.plot(X, y_pred)
# second plot: Cost function
ax2 = plt.subplot(122)
plt.plot(bbs, mses, 'o-')
plt.title('Cost as a function of b')
plt.xlabel('b')
Explanation: Manual exploration of different values of W and b
End of explanation
from keras.models import Sequential
from keras.layers import Dense
from keras.optimizers import Adam, SGD
model = Sequential()
model.add(Dense(1, input_shape=(1,)))
model.summary()
model.compile(Adam(lr = 0.8), loss = 'mean_squared_error')
model.fit(X, y_true, epochs = 40)
y_pred = model.predict(X)
df.plot(kind = 'scatter',
        x = 'Height',
        y = 'Weight',
        title = 'Weight and Height in adults')
plt.plot(X, y_pred, color='red')
W, B = model.get_weights()
W
B
Explanation: Linear Regression with Keras
End of explanation
from sklearn.metrics import r2_score
print("The R2 score is {:0.3f}".format(r2_score(y_true, y_pred)))
Explanation: Evaluating Model Performance
End of explanation
from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(X, y_true,
                                                    test_size=0.2)
len(X_train)
len(X_test)
W[0, 0] = 0.0
B[0] = 0.0
model.set_weights((W, B))
model.fit(X_train, y_train, epochs = 50, verbose = 0)
y_train_pred = model.predict(X_train).ravel()
y_test_pred = model.predict(X_test).ravel()
from sklearn.metrics import mean_squared_error as mse
print("The Mean Squared Error on the Train set is:\t{:0.1f}".format(mse(y_train, y_train_pred)))
print("The Mean Squared Error on the Test set is:\t{:0.1f}".format(mse(y_test, y_test_pred)))
print("The R2 score on the Train set is:\t{:0.3f}".format(r2_score(y_train, y_train_pred)))
print("The R2 score on the Test set is:\t{:0.3f}".format(r2_score(y_test, y_test_pred)))
Explanation: Train Test Split
End of explanation
df = pd.read_csv('./data/user_visit_duration.csv')
df.head()
df.plot(kind = 'scatter', 
        x='Time (min)', 
        y='Buy')
model = Sequential()
model.add(Dense(1, input_shape=(1,), activation='sigmoid'))
model.compile(SGD(lr = 0.5), 
              loss = 'binary_crossentropy', 
              metrics=['accuracy'])
model.summary()
X = df[['Time (min)']].values
y = df['Buy'].values
model.fit(X, y, epochs = 25)
ax = df.plot(kind='scatter', 
             x = 'Time (min)', 
             y ='Buy',
             title = 'Purchase behavior VS time spent on site')
temp = np.linspace(0, 4)
ax.plot(temp, model.predict(temp), color = 'orange')
plt.legend(['model', 'data'])
temp_class = model.predict(temp) > 0.5
ax = df.plot(kind = 'scatter', 
             x = 'Time (min)', 
             y = 'Buy',
             title = 'Purchase behavior VS time spent on site')
temp = np.linspace(0, 4)
ax.plot(temp, temp_class, color = 'orange')
plt.legend(['model', 'data'])
y_pred = model.predict(X)
y_class_pred = y_pred > 0.5
from sklearn.metrics import accuracy_score
print("The accuracy score is {:0.3f}".format(accuracy_score(y, y_class_pred)))
Explanation: Classification
End of explanation
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size = 0.2)
params = model.get_weights()
params = [np.zeros(w.shape) for w in params]
model.set_weights(params)
print("The accuracy score is {:0.3f}".format(accuracy_score(y, model.predict(X) > 0.5)))
model.fit(X_train, y_train, epochs = 25, verbose = 0)
print("The train accuracy score is {:0.3f}".format(accuracy_score(y_train, model.predict(X_train) > 0.5)))
print("The test accuracy score is {:0.3f}".format(accuracy_score(y_test, model.predict(X_test) > 0.5)))
Explanation: Train/Test split
End of explanation
from keras.wrappers.scikit_learn import KerasClassifier
def build_logistic_regression_model():
    model = Sequential()
    model.add(Dense(1, 
                    input_shape = (1,), 
                    activation = 'sigmoid'))
    model.compile(SGD(lr = 0.5),
                  loss = 'binary_crossentropy',
                  metrics = ['accuracy'])
    return model
model = KerasClassifier(build_fn = build_logistic_regression_model,
                        epochs = 25,
                        verbose = 0)
from sklearn.model_selection import cross_val_score, KFold
cv = KFold(3, shuffle = True)
scores = cross_val_score(model, X, y, cv = cv)
scores
print("The cross validation accuracy is {:0.4f} ± {:0.4f}".format(scores.mean(), scores.std()))
Explanation: Cross Validation
End of explanation
from sklearn.metrics import confusion_matrix
confusion_matrix(y, y_class_pred)
def pretty_confusion_matrix(y_true, y_pred, labels = ["False", "True"]):
    cm = confusion_matrix(y_true, y_pred)
    pred_labels = ['Predicted '+ l for l in labels]
    df = pd.DataFrame(cm, index = labels, columns = pred_labels)
    return df
pretty_confusion_matrix(y, y_class_pred, ['Not Buy', 'Buy'])
from sklearn.metrics import precision_score, recall_score, f1_score
print("Precision:\t{:0.3f}".format(precision_score(y, y_class_pred)))
print("Recall:  \t{:0.3f}".format(recall_score(y, y_class_pred)))
print("F1 Score:\t{:0.3f}".format(f1_score(y, y_class_pred)))
from sklearn.metrics import classification_report
print(classification_report(y, y_class_pred))
Explanation: Confusion Matrix
End of explanation
df = pd.read_csv('./data/weight-height.csv')
df.head()
df['Gender'].unique()
pd.get_dummies(df['Gender'], prefix = 'Gender').head()
Explanation: Feature Preprocessing
Categorical Features
End of explanation
df['Height (feet)'] = df['Height']/12.0
df['Weight (100 lbs)'] = df['Weight']/100.0
df.describe().round(2)
Explanation: Feature Transformations
1) Rescale with fixed factor
End of explanation
from sklearn.preprocessing import MinMaxScaler
mms = MinMaxScaler()
df['Weight_mms'] = mms.fit_transform(df[['Weight']])
df['Height_mms'] = mms.fit_transform(df[['Height']])
df.describe().round(2)
Explanation: MinMax normalization
End of explanation
from sklearn.preprocessing import StandardScaler
ss = StandardScaler()
df['Weight_ss'] = ss.fit_transform(df[['Weight']])
df['Height_ss'] = ss.fit_transform(df[['Height']])
df.describe().round(2)
plt.figure(figsize=(15, 5))
for i, feature in enumerate(['Height', 'Height (feet)', 'Height_mms', 'Height_ss']):
    plt.subplot(1, 4, i+1)
    df[feature].plot(kind = 'hist', 
                     title = feature)
    plt.xlabel(feature)
Explanation: 3) Standard normalization
End of explanation
ex1 = pd.read_csv('./data/housing-data.csv')
ex1.head()
ex1.shape
plt.figure(figsize=(20, 5))
for i, feature in enumerate(ex1.columns):
    plt.subplot(1, 4, i + 1)
    ex1[feature].plot(kind = 'hist',
                      title = feature)
    plt.xlabel(feature)
    plt.tight_layout()
X = ex1[['sqft', 'bdrms', 'age']]
X.head()
Y = ex1[['price']]
Y.head()
X = X.values
Y = Y.values
X.shape
Y.shape
from keras.models import Sequential
from keras.layers import Dense
from keras.optimizers import SGD, Adam
model = Sequential()
model.add(Dense(1, input_shape = (3, )))
opt = Adam(lr = 0.8)
model.compile(optimizer = opt, loss = 'mean_squared_error')
model.summary()
from sklearn.model_selection import train_test_split
X_train, X_test, Y_train, Y_test = train_test_split(X, Y, test_size = 0.2)
model.fit(X_train, Y_train, epochs = 10)
from sklearn.metrics import r2_score
Y_train_predicted = model.predict(X_train)
Y_test_predicted = model.predict(X_test)
train_score = r2_score(Y_train, Y_train_predicted)
test_score = r2_score(Y_test, Y_test_predicted)
print('Train set score: \t{:0.3f}'.format(train_score))
print('Test set score: \t{:0.3f}'.format(test_score))
Explanation: Machine Learning Exercises
Exercise 1
You've just been hired at a real estate investment firm and they would like you to build a model for pricing houses. You are given a dataset that contains data for house prices and a few features like number of bedrooms, size in square feet and age of the house. Let's see if you can build a model that is able to predict the price. In this exercise we extend what we have learned about linear regression to a dataset with more than one feature. Here are the steps to complete it:
Load the dataset ../data/housing-data.csv
plot the histograms for each feature
create 2 variables called X and y: X shall be a matrix with 3 columns (sqft,bdrms,age) and y shall be a vector with 1 column (price)
create a linear regression model in Keras with the appropriate number of inputs and output
split the data into train and test with a 20% test size
train the model on the training set and check its accuracy on training and test set
how's your model doing? Is the loss growing smaller?
try to improve your model with these experiments:
normalize the input features with one of the rescaling techniques mentioned above
use a different value for the learning rate of your model
use a different optimizer
once you're satisfied with training, check the R2score on the test set
End of explanation
from sklearn.preprocessing import MinMaxScaler
minmax = MinMaxScaler()
X = minmax.fit_transform(X)
Y = minmax.fit_transform(Y)
X_train, X_test, Y_train, Y_test = train_test_split(X, Y, test_size = 0.2)
model = Sequential()
model.add(Dense(1, input_shape = (3, )))
opt = Adam(lr = 0.8)
model.compile(optimizer = opt, loss = 'mean_squared_error')
model.fit(X_train, Y_train, epochs = 20)
Y_train_predicted = model.predict(X_train)
Y_test_predicted = model.predict(X_test)
train_score = r2_score(Y_train, Y_train_predicted)
test_score = r2_score(Y_test, Y_test_predicted)
print('Train set score: \t{:0.3f}'.format(train_score))
print('Test set score: \t{:0.3f}'.format(test_score))
Explanation: MinMaxScaler
End of explanation
model = Sequential()
model.add(Dense(1, input_shape = (3, )))
opt = Adam(lr = 0.1)
model.compile(optimizer = opt, loss = 'mean_squared_error')
model.fit(X_train, Y_train, epochs = 20, verbose = 1)
Y_train_predicted = model.predict(X_train)
Y_test_predicted = model.predict(X_test)
train_score = r2_score(Y_train, Y_train_predicted)
test_score = r2_score(Y_test, Y_test_predicted)
print('Train set score: \t{:0.3f}'.format(train_score))
print('Test set score: \t{:0.3f}'.format(test_score))
Explanation: Lowering the learning rate
End of explanation
model = Sequential()
model.add(Dense(1, input_shape = (3, )))
opt = SGD(lr = 0.1)
model.compile(optimizer = opt, loss = 'mean_squared_error')
model.fit(X_train, Y_train, epochs = 20)
Y_train_predicted = model.predict(X_train)
Y_test_predicted = model.predict(X_test)
train_score = r2_score(Y_train, Y_train_predicted)
test_score = r2_score(Y_test, Y_test_predicted)
print('Train set score: \t{:0.3f}'.format(train_score))
print('Test set score: \t{:0.3f}'.format(test_score))
Explanation: Using SGD
End of explanation
ex2 = pd.read_csv('./data/HR_comma_sep.csv')
ex2.head()
ex2.info()
ex2.describe()
# Accuracy if predicted that all stay 
# left = 1 : the employee already left
acc = (1 - (ex2['left'].value_counts()[1] / ex2.shape[0])) * 100
print('Predicting all would stay yields accuracy: \t{:0.2f} %'.format(acc))
len(ex2.columns)
ex2['average_montly_hours'].plot(kind = 'hist', 
                                 figsize = (10, 5))
ex2['time_spend_company'].plot(kind = 'hist', 
                                 figsize = (10, 5))
minmax = MinMaxScaler()
ex2['average_montly_hours'] = minmax.fit_transform(ex2['average_montly_hours'].reshape(-1,1))
ex2['time_spend_company'] = minmax.fit_transform(ex2['time_spend_company'].reshape(-1,1))
ex2['average_montly_hours'].plot(kind = 'hist', 
                                 figsize = (10, 5))
# Sales and salary are categorical data
ex2_dummies = pd.get_dummies(ex2[['sales', 'salary']])
ex2_dummies.head()
del ex2['sales'], ex2['salary']
ex2.head()
len(ex2.columns)
data = pd.concat([ex2, ex2_dummies], axis = 1)
data.head()
X = data.ix[:, data.columns != 'left']
X.head()
len(X.columns)
Y = data['left']
from sklearn.model_selection import train_test_split
X = X.values
Y = Y.values
X_train, X_test, Y_train, Y_test = train_test_split(X, Y, test_size = 0.2)
model = Sequential()
model.add(Dense(1, input_dim = X_train.shape[1], activation = 'sigmoid'))
model.compile(optimizer = Adam(lr = 0.05), loss = 'binary_crossentropy', metrics = ['accuracy'])
model.summary()
model.fit(X_train, Y_train, epochs = 20)
Y_test_predicted = model.predict_classes(X_test)
from sklearn.metrics import confusion_matrix, classification_report
def pretty_confusion_matrix(y_true, y_pred, labels=["False", "True"]):
    cm = confusion_matrix(y_true, y_pred)
    pred_labels = ['Predicted '+ l for l in labels]
    df = pd.DataFrame(cm, index = labels, columns = pred_labels)
    return df
pretty_confusion_matrix(Y_test, Y_test_predicted, labels=['Stay', 'Leave'])
print(classification_report(Y_test, Y_test_predicted))
from sklearn.model_selection import cross_val_score, KFold
from keras.wrappers.scikit_learn import KerasClassifier
def build_model():
    model = Sequential()
    model.add(Dense(1, 
                    input_dim = 20, 
                    activation='sigmoid'))
    model.compile(Adam(lr = 0.1), 'binary_crossentropy', metrics=['accuracy'])
    return model
model = KerasClassifier(build_fn = build_model, 
                        epochs = 10, 
                        verbose = 0)
from sklearn.model_selection import KFold, cross_val_score
cross_val = KFold(5, shuffle = True)
scores = cross_val_score(model, X, Y, cv = cross_val)
print("The cross validation accuracy is {:0.4f} ± {:0.4f}".format(scores.mean(), scores.std()))
scores
Explanation: Exercise 2
Your boss was extremely happy with your work on the housing price prediction model and decided to entrust you with a more challenging task. They've seen a lot of people leave the company recently and they would like to understand why that's happening. They have collected historical data on employees and they would like you to build a model that is able to predict which employee will leave next. The would like a model that is better than random guessing. They also prefer false negatives than false positives, in this first phase. Fields in the dataset include:
Employee satisfaction level
Last evaluation
Number of projects
Average monthly hours
Time spent at the company
Whether they have had a work accident
Whether they have had a promotion in the last 5 years
Department
Salary
Whether the employee has left
Your goal is to predict the binary outcome variable left using the rest of the data. Since the outcome is binary, this is a classification problem. Here are some things you may want to try out:
load the dataset at ../data/HR_comma_sep.csv, inspect it with .head(), .info() and .describe().
Establish a benchmark: what would be your accuracy score if you predicted everyone stay?
Check if any feature needs rescaling. You may plot a histogram of the feature to decide which rescaling method is more appropriate.
convert the categorical features into binary dummy columns. You will then have to combine them with the numerical features using pd.concat.
do the usual train/test split with a 20% test size
play around with learning rate and optimizer
check the confusion matrix, precision and recall
check if you still get the same results if you use a 5-Fold cross validation on all the data
Is the model good enough for your boss?
As you will see in this exercise, the a logistic regression model is not good enough to help your boss. In the next chapter we will learn how to go beyond linear models.
This dataset comes from https://www.kaggle.com/ludobenistant/hr-analytics/ and is released under CC BY-SA 4.0 License.
End of explanation
<END_TASK>