<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 CS229
Step1:  A little searching leads us to the Portland housing prices dataset that's used as an example in the lecture. We load the dataset from the CSV file.
Step2:  Let's plot the output variable (the price of a house) against each of the input variables (area in sq. feet, number of bedrooms) to get a little intuition about the data.
Step3:  Let's transform our data into the right matrix format. Note that we add a column of one's to the $X$ matrix, to be multiplied with $\theta_0$.
Step4:  Next we implement the hypothesis and cost functions and the parameter update using gradient descent.
Step5:  We are now ready to fit the model using gradient descent. Let's initialize our parameters to 0 and run 50 iterations of gradient descent to see how it behaves.
Step6:  That doesn't look good. We expected the cost to steadily decrease as gradient descent progressed. Instead, the cost function diverged so much it exceeded our ability to represent it as a floating-point number. What happened?
The answer is that our training data is not on the right scale. The area in square feet is in the thousands, the number of rooms is 1-10 and the price is in the hundreds of thousands. With such widely varying numbers the gradient descent algorithm can overshoot the minimum and instead diverge to infinity.
The solution is to scale all of the variables to have mean of 0 and variance of 1. We can use the scale function from sklearn to do this.
Step7:  We can plot the data again to visualize the effect of the scaling operation.
Step8:  Let us write a function to fit the model such that it automatically stops once the improvement in the value of the cost function is below a certain threshold.
Step9:  Let's try to fit the model again with our scaled input and output matrices.
Step10:  Success!
The gradient descent converges in just 38 steps.
We can verify that our solution is correct by fitting the same data using the library functions for linear regression in sklearn.
Step11:  The parameters are close enough to consider our solution correct.
Stochastic Gradient Descent
<ASSISTANT_TASK:>
Python Code:
%matplotlib inline
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns
import numpy as np
from sklearn.preprocessing import scale
Explanation: CS229: Lecture 2
Linear Regression, Gradient Descent
In this notebook we implement some of the concepts discussed in Lecture 2 of CS229 - Machine Learning.
First we import the relevant Python modules. This notebook has been written assuming Python 3.5
End of explanation
houses = pd.read_csv('house_prices.csv')
Explanation: A little searching leads us to the Portland housing prices dataset that's used as an example in the lecture. We load the dataset from the CSV file.
End of explanation
plt.figure(1)
plt.subplot(211)
plt.xlabel('sq. feet')
plt.ylabel('price (\'000)')
plt.scatter(houses['sqft'], houses['price'])
plt.subplot(212)
plt.xlabel('no. of rooms')
plt.ylabel('price (\'000)')
plt.scatter(houses['rooms'], houses['price'])
plt.tight_layout()
Explanation: Let's plot the output variable (the price of a house) against each of the input variables (area in sq. feet, number of bedrooms) to get a little intuition about the data.
End of explanation
X = houses[['sqft', 'rooms']].as_matrix()
X = np.column_stack([np.ones([X.shape[0]]), X])
y = houses[['price']].as_matrix().ravel()
Explanation: Let's transform our data into the right matrix format. Note that we add a column of one's to the $X$ matrix, to be multiplied with $\theta_0$.
End of explanation
# Hypothesis function
def h(theta, X):
    return np.matmul(X, theta)
# Cost function
def J(theta, X, y):
    d = h(theta, X) - y
    return 0.5 * np.dot(d, d.T)
# One step of gradient descent
def descend(theta, X, y, alpha=0.01):
    error = h(theta, X) - y
    t = theta - alpha * np.matmul(X.T, error)
    return t, np.dot(error, error.T)
Explanation: Next we implement the hypothesis and cost functions and the parameter update using gradient descent.
End of explanation
theta = np.zeros([X.shape[1]])
for i in range(50):
    theta, cost = descend(theta, X, y)
    if i % 10 == 0:
        print("epoch: {0}, cost: {1}".format(i, cost))
print("epoch: {0}, cost: {1}".format(i, cost))
print("theta: {0}".format(theta))
Explanation: We are now ready to fit the model using gradient descent. Let's initialize our parameters to 0 and run 50 iterations of gradient descent to see how it behaves.
End of explanation
X_scaled = scale(X)
y_scaled = scale(y)
Explanation: That doesn't look good. We expected the cost to steadily decrease as gradient descent progressed. Instead, the cost function diverged so much it exceeded our ability to represent it as a floating-point number. What happened?
The answer is that our training data is not on the right scale. The area in square feet is in the thousands, the number of rooms is 1-10 and the price is in the hundreds of thousands. With such widely varying numbers the gradient descent algorithm can overshoot the minimum and instead diverge to infinity.
The solution is to scale all of the variables to have mean of 0 and variance of 1. We can use the scale function from sklearn to do this.
End of explanation
plt.figure(1)
plt.subplot(211)
plt.xlabel('sq. feet')
plt.ylabel('price')
plt.scatter(X_scaled[:, 1], y_scaled)
plt.subplot(212)
plt.xlabel('no. of rooms')
plt.ylabel('price')
plt.scatter(X_scaled[:, 2], y_scaled)
plt.tight_layout()
Explanation: We can plot the data again to visualize the effect of the scaling operation.
End of explanation
def fit(X, y):
    theta = np.zeros([X.shape[1]])
    theta, cost = descend(theta, X, y)
    
    for i in range(10000):
        cost_ = cost
        theta, cost = descend(theta, X, y)
        if cost_ - cost < 1e-7:
            break
            
        if i % 10 == 0:
            print("epoch: {0}, cost: {1}".format(i, cost))
    print("epoch: {0}, cost: {1}".format(i, cost))
    print("theta: {0}".format(theta))
Explanation: Let us write a function to fit the model such that it automatically stops once the improvement in the value of the cost function is below a certain threshold.
End of explanation
fit(X_scaled, y_scaled)
Explanation: Let's try to fit the model again with our scaled input and output matrices.
End of explanation
from sklearn.linear_model import LinearRegression
l = LinearRegression(fit_intercept=False)
l.fit(X_scaled, y_scaled)
l.coef_
Explanation: Success!
The gradient descent converges in just 38 steps.
We can verify that our solution is correct by fitting the same data using the library functions for linear regression in sklearn.
End of explanation
# One step of gradient descent
def stochastic_descend(theta, X, y, alpha=0.01):
    X_sample = np.random.choice(X.shape[0], 47)
    error = h(theta, X[X_sample]) - y[X_sample]
    t = theta - alpha * np.matmul(X[X_sample].T, error)
    return t, np.dot(error, error.T)
X[4]
def stochastic_fit(X, y):
    theta = np.zeros([X.shape[1]])
    theta, cost = descend(theta, X, y)
    
    for i in range(10000):
        cost_ = cost
        theta, cost = descend(theta, X, y)
        if cost_ - cost < 1e-7:
            break
            
        if i % 10 == 0:
            print("epoch: {0}, cost: {1}".format(i, cost))
    print("epoch: {0}, cost: {1}".format(i, cost))
    print("theta: {0}".format(theta))
stochastic_fit(X_scaled, y_scaled)
Explanation: The parameters are close enough to consider our solution correct.
Stochastic Gradient Descent
End of explanation
<END_TASK>