<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 Predicting survival of Titanic Passengers
This notebook explores a dataset containing information of passengers of the Titanic.
The dataset can be downloaded from Kaggle
Tutorial goals
Explore the dataset
Build a simple predictive modeling
Iterate and improve your score
Optional
Step1:  1) Explore the dataset
Numerical exploration
Load the csv file into memory using Pandas
Describe each attribute
is it discrete?
is it continuous?
is it a number?
is it text?
Identify the target
Check if any values are missing
Load the csv file into memory using Pandas
Step2:  What's the content of df ?
Step3:  Describe each attribute (is it discrete? is it continuous? is it a number? is it text?)
Step4:  Is Pclass a continuous or discrete class?
Step5:  What about these
Step6:  and what about these
Step7:  Identify the target
What are we trying to predict?
ah, yes... Survival!
Step8:  Check if any values are missing
Step9:  Mental notes so far
Step10:  impute the missing values for Age using the median Age
Step11:  check the influence of Age
Step12:  Check the influence of Sex on Survival
Step13:  Check the influence of Pclass on Survival
Step14:  Ok, so, Age and Pclass seem to have some influence on survival rate.
Let's build a simple model to test that
Define a new feature called "Male" that is 1 if Sex = 'male' and 0 otherwise
Step15:  Define simplest model as benchmark
The simplest model is a model that predicts 0 for everybody, i.e. no survival.
How good is it?
Step16:  We need to do better than that
Define features (X) and target (y) variables
Step17:  Initialize a decision tree model
Step18:  Split the features and the target into a Train and a Test subsets.
Ratio should be 80/20
Step19:  Train the model
Step20:  Calculate the model score
Step21:  Print the confusion matrix for the decision tree model
<ASSISTANT_TASK:>
Python Code:
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline
Explanation: Predicting survival of Titanic Passengers
This notebook explores a dataset containing information of passengers of the Titanic.
The dataset can be downloaded from Kaggle
Tutorial goals
Explore the dataset
Build a simple predictive modeling
Iterate and improve your score
Optional: upload your prediction to Kaggle using the test dataset
How to follow along:
git clone https://github.com/Dataweekends/odsc_intro_to_data_science
cd odsc_intro_to_data_science
ipython notebook
We start by importing the necessary libraries:
End of explanation
df = pd.read_csv('titanic-train.csv')
Explanation: 1) Explore the dataset
Numerical exploration
Load the csv file into memory using Pandas
Describe each attribute
is it discrete?
is it continuous?
is it a number?
is it text?
Identify the target
Check if any values are missing
Load the csv file into memory using Pandas
End of explanation
df.head(3)
Explanation: What's the content of df ?
End of explanation
df.info()
Explanation: Describe each attribute (is it discrete? is it continuous? is it a number? is it text?)
End of explanation
df['Pclass'].value_counts()
Explanation: Is Pclass a continuous or discrete class?
End of explanation
df['SibSp'].value_counts()
df['Parch'].value_counts()
Explanation: What about these: ('SibSp', 'Parch')?
End of explanation
df[['Ticket', 'Fare', 'Cabin']].head(3)
df['Embarked'].value_counts()
Explanation: and what about these: ('Ticket', 'Fare', 'Cabin', 'Embarked')?
End of explanation
df['Survived'].value_counts()
Explanation: Identify the target
What are we trying to predict?
ah, yes... Survival!
End of explanation
df.info()
Explanation: Check if any values are missing
End of explanation
df['Age'].plot(kind='hist', figsize=(10,6))
plt.title('Distribution of Age', size = '20')
plt.xlabel('Age', size = '20')
plt.ylabel('Number of passengers', size = '20')
median_age = df['Age'].median()
plt.axvline(median_age, color = 'r')
median_age
Explanation: Mental notes so far:
Dataset contains 891 entries
1 Target column (Survived)
11 Features:
6 numerical, 5 text
1 useless (PassengerId)
3 categorical (Pclass, Sex, Embarked)
4 numerical, > 0 (Age, SibSp, Parch, Fare)
3 not sure how to treat (Name, Ticket, Cabin)
Age is only available for 714 passengers
Cabin is only available for 204 passengers
Embarked is missing for 2 passengers
Visual exploration
plot the distribution of Age
impute the missing values for Age using the median Age
check the influence of Age, Sex and Class on Survival
Plot the distribution of Age
End of explanation
df['Age'].fillna(median_age, inplace = True)
df.info()
Explanation: impute the missing values for Age using the median Age
End of explanation
df[df['Survived']==1]['Age'].plot(kind='hist', bins = 10, range = (0,100), figsize=(10,6), alpha = 0.3, color = 'g')
df[df['Survived']==0]['Age'].plot(kind='hist', bins = 10, range = (0,100), figsize=(10,6), alpha = 0.3, color = 'r')
plt.title('Distribution of Age', size = '20')
plt.xlabel('Age', size = '20')
plt.ylabel('Number of passengers', size = '20')
plt.legend(['Survived', 'Dead'])
plt.show()
Explanation: check the influence of Age
End of explanation
survival_by_gender = df[['Sex','Survived']].pivot_table(columns =
                        ['Survived'], index = ['Sex'], aggfunc=len)
survival_by_gender
survival_by_gender.plot(kind = 'bar', stacked = True)
plt.show()
Explanation: Check the influence of Sex on Survival
End of explanation
survival_by_Pclass = df[['Pclass','Survived']].pivot_table(columns =
                        ['Survived'], index = ['Pclass'], aggfunc=len)
survival_by_Pclass
survival_by_Pclass.plot(kind = 'bar', stacked = True)
plt.show()
Explanation: Check the influence of Pclass on Survival
End of explanation
df['Male'] = df['Sex'].map({'male': 1, 'female': 0})
df[['Sex', 'Male']].head()
Explanation: Ok, so, Age and Pclass seem to have some influence on survival rate.
Let's build a simple model to test that
Define a new feature called "Male" that is 1 if Sex = 'male' and 0 otherwise
End of explanation
actual_dead = len(df[df['Survived'] == 0])
total_passengers = len(df)
ratio_of_dead = actual_dead / float(total_passengers)
print "If I predict everybody dies, I'm correct %0.1f %% of the time" % (100 * ratio_of_dead)
df['Survived'].value_counts()
Explanation: Define simplest model as benchmark
The simplest model is a model that predicts 0 for everybody, i.e. no survival.
How good is it?
End of explanation
X = df[['Male', 'Pclass', 'Age']]
y = df['Survived']
Explanation: We need to do better than that
Define features (X) and target (y) variables
End of explanation
from sklearn.tree import DecisionTreeClassifier
model = DecisionTreeClassifier(random_state=0)
model
Explanation: Initialize a decision tree model
End of explanation
from sklearn.cross_validation import train_test_split
X_train, X_test, y_train, y_test = train_test_split(X, y, 
                            test_size = 0.2, random_state=0)
Explanation: Split the features and the target into a Train and a Test subsets.
Ratio should be 80/20
End of explanation
model.fit(X_train, y_train)
Explanation: Train the model
End of explanation
my_score = model.score(X_test, y_test)
print "Classification Score: %0.2f" % my_score
Explanation: Calculate the model score
End of explanation
from sklearn.metrics import confusion_matrix
y_pred = model.predict(X_test)
print "\n=======confusion matrix=========="
print confusion_matrix(y_test, y_pred)
Explanation: Print the confusion matrix for the decision tree model
End of explanation
<END_TASK>