<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 Gradient descent for machine learning - a quick introduction
In this notebook we are going to use gradient descent to estimate the parameters of a model. In this case we are going to compute the parameters to convert temperatures fom Farenheit to Kelvin.
Given that we are approaching this from a machine learning perspective, we are going to determine our scaling factor and offset value by gradient descent, given some example temperatures on both scales. In other words, we are going to learn the parameters from the data.
First, some imports
Step1:  Our data set
Step2:  Show our data set in a table
Step3:  Model - linear regression
Temperatures can be converted using a linear model of the form $y=ax+b$.
$x$ and $y$ are samples in our dataset; $X={x_0...X_N}$ and $Y={y_0...Y_N}$ while $a$ and $b$ are the model parameters.
Initialise model
Lets try initialising the parameters $a$ randomly and $b$ to 0 and see what it predicts
Step4:  How good is our guess?
To estimate the accuracy of our model, lets compute the squared error. We use the squared error since its value will always be positive and larger errors will have a greater cost due to being squared
Step7:  Reducing the error
We reduce the error by taking the gradient of the squared error with respect to the parameters $a$ and $b$ and iteratively modifying the values of $a$ and $b$ in the direction of the negated gradient.
Lets determine the expressions for the gradient of the squared error $\epsilon$ with respect to $a$ and $b$
Step8:  Define learning rate and show initial state
Step9:  Gradient descent
Run this cell repeatedly to see gradient descent in action
<ASSISTANT_TASK:>
Python Code:
import numpy as np
import pandas as pd
Explanation: Gradient descent for machine learning - a quick introduction
In this notebook we are going to use gradient descent to estimate the parameters of a model. In this case we are going to compute the parameters to convert temperatures fom Farenheit to Kelvin.
Given that we are approaching this from a machine learning perspective, we are going to determine our scaling factor and offset value by gradient descent, given some example temperatures on both scales. In other words, we are going to learn the parameters from the data.
First, some imports:
End of explanation
INDEX = ['Boiling point of He',
         'Boiling point of N',
         'Melting point of H2O',
         'Body temperature',
         'Boiling point of H2O']
X = np.array([-452.1, -320.4, 32.0, 98.6, 212.0])
Y = np.array([4.22, 77.36, 273.2, 310.5, 373.2])
Explanation: Our data set:
End of explanation
pd.DataFrame(np.stack([X, Y]).T, index=INDEX,
             columns=['Fahrenheit ($x$)', 'Kelvin ($y$)'])
Explanation: Show our data set in a table:
End of explanation
# Lets initialise `a` to between 1.0 and 2.0; it is therefore impossible
# for it to choose a (nearly) correct value at the start, forcing our model to do some work.
a = np.random.uniform(1.0, 2.0, size=())
b = 0.0
print('a={}, b={}'.format(a, b))
Y_pred = X * a + b
pd.DataFrame(np.stack([X, Y, Y_pred]).T, index=INDEX,
             columns=['Fahrenheit ($x$)', 'Kelvin ($y$)', '$y_{pred}$'])
Explanation: Model - linear regression
Temperatures can be converted using a linear model of the form $y=ax+b$.
$x$ and $y$ are samples in our dataset; $X={x_0...X_N}$ and $Y={y_0...Y_N}$ while $a$ and $b$ are the model parameters.
Initialise model
Lets try initialising the parameters $a$ randomly and $b$ to 0 and see what it predicts:
End of explanation
sqr_err = (Y_pred - Y)**2
pd.DataFrame(np.stack([X, Y, Y_pred, sqr_err]).T, index=INDEX,
             columns=['Fahrenheit ($x$)', 'Kelvin ($y$)', '$y_{pred}$', 'squared err ($\epsilon$)'])
Explanation: How good is our guess?
To estimate the accuracy of our model, lets compute the squared error. We use the squared error since its value will always be positive and larger errors will have a greater cost due to being squared:
End of explanation
def iterative_gradient_descent_step(a, b, lr):
    
    A single gradient descent iteration
    
    :param a: current value of `a`
    :param b: current value of `b`
    :param lr: learning rate
    
    :return: a tuple `(a_next, b_next)` that are the values of `a` and `b` after the iteration.
    
    # Derivative of a and b w.r.t. epsilon:
    da_depsilon = (2 * a * X**2 + 2 * b * X - 2 * X * Y).mean()
    db_depsilon = (2 * b + 2 * a * X - 2 * Y).mean()
    
    # Gradient descent:
    a = a - da_depsilon * lr
    b = b - db_depsilon * lr
    
    # Return new values
    return a, b
def state_as_table(a, b):
    
    Helper function to generate a Pandas DataFrame showing the current state, including predicted values and errors
    
    :param a: current value of `a`
    :param b: current value of `b`
    
    :return: tuple `(df, mean_sqr_err)` where `df` is the Pandas DataFrame and `sqr_err` is the mean squared error
    
    Y_pred = X * a + b
    sqr_err = (Y_pred - Y)**2
    df = pd.DataFrame(np.stack([X, Y, Y_pred, sqr_err]).T, index=INDEX,
                      columns=['Fahrenheit ($x$)', 'Kelvin ($y$)', '$y_{pred}$', 'squared err ($\epsilon$)'])    
    return df, sqr_err.mean()
Explanation: Reducing the error
We reduce the error by taking the gradient of the squared error with respect to the parameters $a$ and $b$ and iteratively modifying the values of $a$ and $b$ in the direction of the negated gradient.
Lets determine the expressions for the gradient of the squared error $\epsilon$ with respect to $a$ and $b$:
$\epsilon_i = (ax_i + b - y_i)^2 = a^2x_i^2 + 2abx_i - 2ax_iy_i + b^2 + y_i^2 - 2by_i$
In terms of $a$: $\epsilon_i = a^2x_i^2 + a(2bx_i - 2x_iy_i) + b^2 + y_i^2 - 2by_i$
So ${d\epsilon_i\over{da}} = 2ax_i^2 + 2bx_i - 2x_iy_i$
In terms of $b$: $\epsilon = b^2 + b(2ax_i - 2y_i) + a^2x_i^2- 2ax_iy_i - 2by_i$
So ${d\epsilon_i\over{db}} = 2b + 2ax_i - 2y_i$
The above expressions apply to single samples only. To apply them to all of our 5 data points, we need to use the mean squared error. The mean squared error is the sum of the individual errors divided by the number of data points $N$. The derivative of the mean squared error w.r.t. $a$ and $b$ will also be the sum of the individual derivatives, divided by $N$.
Gradient descent
Gradient descent is performed iteratively; each parameter is modified independently as so:
$a' = a - \gamma {d\epsilon_i\over{da}}$
$b' = b - \gamma {d\epsilon_i\over{db}}$
where $\gamma$ is the learning rate.
Implementation
We now have all we need to define some gradient descent helper functions:
End of explanation
LEARNING_RATE = 0.00001
N_ITERATIONS = 50000
df, mean_sqr_err = state_as_table(a, b)
print('a = {}, b = {}, mean sqr. err. = {}'.format(a, b, mean_sqr_err))
df
Explanation: Define learning rate and show initial state:
End of explanation
for i in xrange(N_ITERATIONS):
    a, b = iterative_gradient_descent_step(a, b, LEARNING_RATE)
df, mean_sqr_err = state_as_table(a, b)
print('a = {}, b = {}, mean sqr. err. = {}'.format(a, b, mean_sqr_err))
df
Explanation: Gradient descent
Run this cell repeatedly to see gradient descent in action:
End of explanation
<END_TASK>