<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 Logistic Regression
Resources
Step1:  The logistic regression equation has a very simiar representation like linear regression. The difference is that the output value being modelled is binary in nature.
$$\hat{y}=\frac{e^{\beta_0+\beta_1x_1}}{1+\beta_0+\beta_1x_1}$$
or
$$\hat{y}=\frac{1.0}{1.0+e^{-\beta_0-\beta_1x_1}}$$
$\beta_0$ is the intecept term
$\beta_1$ is the coefficient for $x_1$
$\hat{y}$ is the predicted output with real value between 0 and 1. To convert this to binary output of 0 or 1, this would either need to be rounded to an integer value or a cutoff point be provided to specify the class segregation point.
Step2:  Making Predictions with Logistic Regression
$$\hat{y}=\frac{1.0}{1.0+e^{-\beta_0-\beta_1x_i}}$$
$\beta_0$ is the intecept term
$\beta_1$ is the coefficient for $x_i$
$\hat{y}$ is the predicted output with real value between 0 and 1. To convert this to binary output of 0 or 1, this would either need to be rounded to an integer value or a cutoff point be provided to specify the class segregation point.
Step3:  Let's say you have been provided with the coefficient
Step4:  Learning the Logistic Regression Model
The coefficients (Beta values b) of the logistic regression algorithm must be estimated from your training data. This is done using maximum-likelihood estimation.
Maximum-likelihood estimation is a common learning algorithm used by a variety of machine learning algorithms, although it does make assumptions about the distribution of your data (more on this when we talk about preparing your data).
The best coefficients would result in a model that would predict a value very close to 1 (e.g. male) for the default class and a value very close to 0 (e.g. female) for the other class. The intuition for maximum-likelihood for logistic regression is that a search procedure seeks values for the coefficients (Beta values) that minimize the error in the probabilities predicted by the model to those in the data (e.g. probability of 1 if the data is the primary class).
We are not going to go into the math of maximum likelihood. It is enough to say that a minimization algorithm is used to optimize the best values for the coefficients for your training data. This is often implemented in practice using efficient numerical optimization algorithm (like the Quasi-newton method).
When you are learning logistic, you can implement it yourself from scratch using the much simpler gradient descent algorithm.
Learning with Stochastic Gradient Descent
Logistic Regression uses gradient descent to update the coefficients.
Each gradient descent iteration, the coefficients are updated using the equation
<ASSISTANT_TASK:>
Python Code:
import numpy as np
import matplotlib.pyplot as plt
import seaborn
%matplotlib inline
x = np.linspace(-6, 6, num = 1000)
plt.figure(figsize = (12,8))
plt.plot(x, 1 / (1 + np.exp(-x))); # Sigmoid Function
plt.title("Sigmoid Function");
Explanation: Logistic Regression
Resources:
Logistic Regression Tutorial for Machine Learning
Logistic Regression for Machine Learning
How To Implement Logistic Regression With Stochastic Gradient Descent From Scratch With Python
Logistic regression is the go-to linear classification algorithm for two-class problems. It is easy to implement, easy to understand and gets great results on a wide variety of problems, even when the expectations the method has for your data are violated.
Description
Logistic Regression
Logistic regression is named for the function used at the core of the method, the logistic function.
The logistic function, also called the Sigmoid function was developed by statisticians to describe properties of population growth in ecology, rising quickly and maxing out at the carrying capacity of the environment. It’s an S-shaped curve that can take any real-valued number and map it into a value between 0 and 1, but never exactly at those limits.
$$\frac{1}{1 + e^{-x}}$$
$e$ is the base of the natural logarithms and $x$ is value that you want to transform via the logistic function.
End of explanation
tmp = [0, 0.4, 0.6, 0.8, 1.0]
tmp
np.round(tmp)
np.array(tmp) > 0.7
Explanation: The logistic regression equation has a very simiar representation like linear regression. The difference is that the output value being modelled is binary in nature.
$$\hat{y}=\frac{e^{\beta_0+\beta_1x_1}}{1+\beta_0+\beta_1x_1}$$
or
$$\hat{y}=\frac{1.0}{1.0+e^{-\beta_0-\beta_1x_1}}$$
$\beta_0$ is the intecept term
$\beta_1$ is the coefficient for $x_1$
$\hat{y}$ is the predicted output with real value between 0 and 1. To convert this to binary output of 0 or 1, this would either need to be rounded to an integer value or a cutoff point be provided to specify the class segregation point.
End of explanation
dataset = [[-2.0011, 0],
           [-1.4654, 0],
           [0.0965, 0],
           [1.3881, 0],
           [3.0641, 0],
           [7.6275, 1],
           [5.3324, 1],
           [6.9225, 1],
           [8.6754, 1],
           [7.6737, 1]]
Explanation: Making Predictions with Logistic Regression
$$\hat{y}=\frac{1.0}{1.0+e^{-\beta_0-\beta_1x_i}}$$
$\beta_0$ is the intecept term
$\beta_1$ is the coefficient for $x_i$
$\hat{y}$ is the predicted output with real value between 0 and 1. To convert this to binary output of 0 or 1, this would either need to be rounded to an integer value or a cutoff point be provided to specify the class segregation point.
End of explanation
coef = [-0.806605464, 0.2573316]
for row in dataset:
    yhat = 1.0 / (1.0 + np.exp(- coef[0] - coef[1] * row[0]))
    print("yhat {0:.4f}, yhat {1}".format(yhat, round(yhat)))
Explanation: Let's say you have been provided with the coefficient
End of explanation
from sklearn.linear_model import LogisticRegression
dataset
X = np.array(dataset)[:, 0:1]
y = np.array(dataset)[:, 1]
X
y
clf_LR = LogisticRegression(C=1.0, penalty='l2', tol=0.0001)
clf_LR.fit(X,y)
clf_LR.predict(X)
clf_LR.predict_proba(X)
Explanation: Learning the Logistic Regression Model
The coefficients (Beta values b) of the logistic regression algorithm must be estimated from your training data. This is done using maximum-likelihood estimation.
Maximum-likelihood estimation is a common learning algorithm used by a variety of machine learning algorithms, although it does make assumptions about the distribution of your data (more on this when we talk about preparing your data).
The best coefficients would result in a model that would predict a value very close to 1 (e.g. male) for the default class and a value very close to 0 (e.g. female) for the other class. The intuition for maximum-likelihood for logistic regression is that a search procedure seeks values for the coefficients (Beta values) that minimize the error in the probabilities predicted by the model to those in the data (e.g. probability of 1 if the data is the primary class).
We are not going to go into the math of maximum likelihood. It is enough to say that a minimization algorithm is used to optimize the best values for the coefficients for your training data. This is often implemented in practice using efficient numerical optimization algorithm (like the Quasi-newton method).
When you are learning logistic, you can implement it yourself from scratch using the much simpler gradient descent algorithm.
Learning with Stochastic Gradient Descent
Logistic Regression uses gradient descent to update the coefficients.
Each gradient descent iteration, the coefficients are updated using the equation:
$$\beta=\beta+\textrm{learning rate}\times (y-\hat{y}) \times \hat{y} \times (1-\hat{y}) \times x $$
Using Scikit Learn to Estimate Coefficients
End of explanation
<END_TASK>