<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 Logistic Regression (Classification)
So far we have been looking at regression problems, where the solution are continous variables. Let us look at a binary classification problem - where we need to identify the correct class [0 or 1]
Step1:  Lets say we want to classify the vehicles by 'Hatchback' and 'Sedan'
Step2:  Why Linear Function does not work
Now we can use a linear function and try to choose a cutoff value to show where the two class separate - lets say cutoff = 0.5
Step3:  However, there are two problems with this approach
- The cut-off value is highly influenced by outliers
- The value of linear regression is unbounded, while our classification is bounded
Step4:  Logistic Function
So we need a function which is bounded between 0 < f(X) < 1, for our classification to work. We will use a logit function for this purpose
$$ P(z) = \frac {1}{1 + e^{-z}} $$
Step5:  So now we can transpose our linear regression problem with this logit function
$$ y = X\beta $$
becomes
$$ P(y) = P(X\beta) $$
$$ y = 0, when \, P(y) >0.5 $$ 
which is equivalent to $$X\beta > 0 $$
Lets see this in the example in 'price' and 'kmpl' variables, for class $types$
$$ \beta_0 + \beta_1price + \beta_2kmpl + \epsilon > 0 $$
So we are looking for a line which will fulfill the above condition
Step6:  Cost Function
$$Cost (P(X\beta), y)) = 
\begin{cases}
-log(P(X\beta)),  & \text{if $y=1$ } \
-log(1 -P(X\beta)), & \text{if $y =0$}
\end{cases}$$
Step7:  Gradient Descent for Logistic Function
To make it easier to work with we can write this in one line - Log-Likelihood Function
$$ C(\beta) = \frac{1}{n} (- y^T * log(P(X\beta)) - (1- y^T) * log(1 -P(X\beta))) $$
If we were to differentiate this, we will get our gradient which is very similiar to our linear regression one
<ASSISTANT_TASK:>
Python Code:
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
%matplotlib inline
plt.style.use('fivethirtyeight')
plt.rcParams['figure.figsize'] = (10, 6)
pop = pd.read_csv('data/cars_small.csv')
pop.head()
Explanation: Logistic Regression (Classification)
So far we have been looking at regression problems, where the solution are continous variables. Let us look at a binary classification problem - where we need to identify the correct class [0 or 1]
End of explanation
class_mapping = {'Hatchback': 0, 'Sedan': 1}
pop['types'] = pop['type'].map(class_mapping)
plt.scatter(pop.price, pop.types, c=pop.types, s = 150, alpha = 0.8 )
plt.xlabel('price')
plt.ylabel('types')
Explanation: Lets say we want to classify the vehicles by 'Hatchback' and 'Sedan'
End of explanation
def ols (df, xlabel, ylabel):
    n = df.shape[0]
    x0 = np.ones(n)
    x1 = df[xlabel]
    X = np.c_[x0, x1]
    X = np.asmatrix(X)
    y = np.transpose(np.asmatrix(df[ylabel]))
    X_T = np.transpose(X)
    X_pseudo = np.linalg.inv(X_T * X) * X_T
    beta = X_pseudo * y
    return beta
def plot_ols(df, xlabel, ylabel):
    beta = ols(df, 'price', 'types')
    beta_0 = beta.item(0)
    beta_1 = beta.item(1)
    plt.scatter(df[xlabel], df[ylabel], c=df[ylabel], s = 150, alpha = 0.8 )
    plt.xlabel(xlabel)
    plt.ylabel(ylabel)
    
    y = beta_0 + beta_1 * df[xlabel]
    plt.plot(df[xlabel], y, '-')
    
    cutoff = (0.5 - beta_0)/beta_1
    plt.vlines(cutoff, -0.4, 1.4)
ols(pop, 'price', 'types')
plot_ols(pop, 'price', 'types')
Explanation: Why Linear Function does not work
Now we can use a linear function and try to choose a cutoff value to show where the two class separate - lets say cutoff = 0.5
End of explanation
pop1 = pop.copy()
pop1.tail()
# Lets create an outlier
pop1.loc[37,'price'] = 1500
pop1.loc[41,'price'] = 2000
plot_ols(pop1, 'price', 'types')
Explanation: However, there are two problems with this approach
- The cut-off value is highly influenced by outliers
- The value of linear regression is unbounded, while our classification is bounded
End of explanation
z = np.linspace(-10, 10, 100)
p = 1/(1+np.exp(-z))
plt.plot(z,p)
plt.hlines(0.5, -20,20)
plt.vlines(0, 0,1)
plt.xlabel('z')
plt.ylabel('P(z)')
Explanation: Logistic Function
So we need a function which is bounded between 0 < f(X) < 1, for our classification to work. We will use a logit function for this purpose
$$ P(z) = \frac {1}{1 + e^{-z}} $$
End of explanation
plt.scatter(pop['kmpl'], pop['price'], c=pop['types'], s = 150, alpha = 0.8 )
plt.xlabel('kmpl')
plt.ylabel('price')
Explanation: So now we can transpose our linear regression problem with this logit function
$$ y = X\beta $$
becomes
$$ P(y) = P(X\beta) $$
$$ y = 0, when \, P(y) >0.5 $$ 
which is equivalent to $$X\beta > 0 $$
Lets see this in the example in 'price' and 'kmpl' variables, for class $types$
$$ \beta_0 + \beta_1price + \beta_2kmpl + \epsilon > 0 $$
So we are looking for a line which will fulfill the above condition
End of explanation
z = np.linspace(0.001, 0.999, 1000)
c1 = -np.log(z)
c2 = -np.log(1-z)
plt.plot(z,c1)
plt.plot(z,c2)
#plt.hlines(0.5, -10,10)
#plt.vlines(0, 0,1)
plt.xlabel('z')
plt.ylabel('Cost')
Explanation: Cost Function
$$Cost (P(X\beta), y)) = 
\begin{cases}
-log(P(X\beta)),  & \text{if $y=1$ } \
-log(1 -P(X\beta)), & \text{if $y =0$}
\end{cases}$$
End of explanation
n = pop.shape[0]
x0 = np.ones(n)
x1 = pop.kmpl
x2 = pop.price
X_actual = np.c_[x1, x2]
X_norm = (X_actual - np.mean(X_actual, axis=0)) / np.std(X_actual, axis=0)
X = np.c_[x0, X_norm]
X = np.asmatrix(X)
y = np.asmatrix(pop.types.values.reshape(-1,1))
b = np.asmatrix([[0],[0],[0]])
def P(z): 
    return 1.0/(1+np.exp(-z))
def cost(X,y,b,n):
    C = (- y.T*np.log(P(X*b))-(1-y.T)*np.log(1-P(X*b)))/n
    return C[0,0]
def gradient(X,y,b,n):
    g = (2/n)*X.T*(P(X*b) - y)
    return g
def gradient_descent_logistic (eta, epochs, X, y, n):
    
    # Set Initial Values 
    b = np.asmatrix([[0],[0],[0]])
    c = cost(X,y,b,n)
    c_all = []
    c_all.append(c)
    # Run the calculation for those many epochs
    for i in range(epochs):
        g = gradient(X,y,b,n)
        b = b - eta * g
        c = cost(X,y,b,n)
        c_all.append(c)
            
    return c_all, b
from __future__ import division
x1_min, x1_max = -3, 3
x2_min, x2_max = -3, 3
xx1, xx2 = np.meshgrid(np.arange(x1_min, x1_max, (x1_max - x1_min)/100), 
                       np.arange(x2_min, x2_max, (x2_max - x2_min)/100))
xx = np.c_[np.ones(xx1.ravel().shape[0]), xx1.ravel(), xx2.ravel()]
def plot_gradient_descent(eta, epoch, gradient_func):
    
    es, bs = gradient_func(eta, epoch, X, y, n)
    
    # Plot the intercept and coefficients
    plt.subplot(1, 2, 1)
    #plt.tight_layout()
    
    # Plot the probabilty plot contour
    Z = P(xx*bs)
    Z = Z.reshape(xx1.shape)
    cs = plt.contourf(xx1, xx2, Z, cmap=plt.cm.viridis, alpha = 0.5)
    plt.colorbar(cs)
    
    # Plot the intercept and coefficients
    plt.scatter(X[:,1], X[:,2], c=pop.types, s = 150, alpha = 0.8 )
    plt.xlabel('kmpl')
    plt.ylabel('price')
    
    # Plot the error rates
    plt.subplot(1, 2, 2)
    plt.plot(es)
    plt.xlabel('Epochs')
    plt.ylabel('Error')
plot_gradient_descent(0.05, 1000, gradient_descent_logistic)
Explanation: Gradient Descent for Logistic Function
To make it easier to work with we can write this in one line - Log-Likelihood Function
$$ C(\beta) = \frac{1}{n} (- y^T * log(P(X\beta)) - (1- y^T) * log(1 -P(X\beta))) $$
If we were to differentiate this, we will get our gradient which is very similiar to our linear regression one:
$$ \nabla C(\beta) =  \frac {2}{n} X^T(P(X\beta)−y) $$
and are gradient descent algorithm will be 
$$ \beta_{i+1} = \beta_{i} - \eta * \nabla C(\beta)$$
End of explanation
<END_TASK>