<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 Exercise 05
Logistic regression exercise with Titanic data
We'll be working with a dataset from Kaggle's Titanic competition
Step1:  Create X and y
Define Pclass and Parch as the features, and Survived as the response.
Step2:  Exercise 5.1
Split the data into training and testing sets
Step3:  Exercise 5.2
Fit a logistic regression model and examine the coefficients
Confirm that the coefficients make intuitive sense.
Step4:  Exercise 5.3
Make predictions on the testing set and calculate the accuracy
Step5:  Exercise 5.4
Confusion matrix of Titanic predictions
Step6:  Exercise 5.5
Increase sensitivity by lowering the threshold for predicting survival
Create a new classifier by changing the probability threshold to 0.3
What is the new confusion matrix?
<ASSISTANT_TASK:>
Python Code:
import pandas as pd
url = 'https://raw.githubusercontent.com/justmarkham/DAT8/master/data/titanic.csv'
titanic = pd.read_csv(url, index_col='PassengerId')
titanic.head()
Explanation: Exercise 05
Logistic regression exercise with Titanic data
We'll be working with a dataset from Kaggle's Titanic competition: data, data dictionary
Goal: Predict survival based on passenger characteristics
The sinking of the RMS Titanic is one of the most infamous shipwrecks in history.  On April 15, 1912, during her maiden voyage, the Titanic sank after colliding with an iceberg, killing 1502 out of 2224 passengers and crew. This sensational tragedy shocked the international community and led to better safety regulations for ships.
One of the reasons that the shipwreck led to such loss of life was that there were not enough lifeboats for the passengers and crew. Although there was some element of luck involved in surviving the sinking, some groups of people were more likely to survive than others, such as women, children, and the upper-class.
In this challenge, we ask you to complete the analysis of what sorts of people were likely to survive. In particular, we ask you to apply the tools of machine learning to predict which passengers survived the tragedy.
Read the data into Pandas
End of explanation
feature_cols = ['Pclass', 'Parch']
X = titanic[feature_cols]
Y = titanic.Survived
Explanation: Create X and y
Define Pclass and Parch as the features, and Survived as the response.
End of explanation
import numpy as np
# Insert code here
random_sample = np.random.rand(y.shape[0])
X_train, X_test = X[random_sample<0.7], X[random_sample>=0.7]
Y_train, Y_test = Y[random_sample<0.7], Y[random_sample>=0.7]
print(Y_train.shape, Y_test.shape)
import numpy as np
from sklearn.cross_validation import train_test_split
X_train, X_test, Y_train, Y_test = train_test_split(X, Y, test_size=0.33, random_state=42)
print(Y_train.shape, Y_test.shape)
Explanation: Exercise 5.1
Split the data into training and testing sets
End of explanation
from sklearn.linear_model import LogisticRegression
logreg = LogisticRegression(C=1e9)
mod1=logreg.fit(X_train, Y_train)
mod1.coef_
Explanation: Exercise 5.2
Fit a logistic regression model and examine the coefficients
Confirm that the coefficients make intuitive sense.
End of explanation
titanic['survive_pred'] = logreg.predict(X)
titanic.head()
survive_pred = logreg.predict(X_test)
(Y_test == survive_pred).mean()
Explanation: Exercise 5.3
Make predictions on the testing set and calculate the accuracy
End of explanation
from sklearn.metrics import confusion_matrix
confusion_matrix(Y_test, survive_pred)
Explanation: Exercise 5.4
Confusion matrix of Titanic predictions
End of explanation
survive_pred_prob=logreg.predict_proba(X_test)[:,1]
predict2=np.where(survive_pred_prob >= 0.7, 1, 0)
confusion_matrix(Y_test, predict2)
(Y_test == predict2).mean()
survive_pred_prob=logreg.predict_proba(X_test)[:,1]
predict2=np.where(survive_pred_prob >= 0.3, 1, 0)
confusion_matrix(Y_test, predict2)
(Y_test == predict2).mean()
Explanation: Exercise 5.5
Increase sensitivity by lowering the threshold for predicting survival
Create a new classifier by changing the probability threshold to 0.3
What is the new confusion matrix?
End of explanation
<END_TASK>