<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 P2 - Titanic Data
<hr>
Questions
What are the most important factors related to the probability of survival ?
<hr>
Load Data
Step1:  Name and Embarked are dropped from the dataset because passenger name and embarking location shouldn't have any meaningful correlation with their chance of surviving. Arguably, the embarking locations might give some indication of passengers' social-economical background. However, I will be in favor of using "Pclass",  because it is specifically mentioned in the special notes in the data souce to be a proxy for social-economic status. 
I dropped Fare in favor of Pclass for the same reason. 
Cabin is an interesting feature. It is plausible the exact location of the passenger when the accident happens have an significant impact on the chance of survival. According to Wikipedia, the Titanic hits an iceburg at 11
Step2:  <hr>
Data Exploration
Visualisation
We first use the pointplot to explore the effect of "Sex", "Pclass", "SibSp", and "Parch" on survival. The pointplot is great for categorical variable and numerical variable with small countable range. It provides the mean estimate and confidence interval for each possible value of the variable. 
For age, we will compare the kdeplot for both who survived and those didn't survive. If the kde are similar, then it indicates the age have little correlation with the chance of survival. If one kde value for survival is higher than the other at a particular age range, then it suggests the chance of survival is higher for than age group.
Step3:  Sex
The deep slope and small verticle bars (representing 99% confidence interval of mean) in the first visualisation suggests that "Sex" is strongly correlated with the chance of survival. Females have much higher chance of survival than male. 
Social class (Pclass)
Similar to "Sex", the second visualisation shows people with higher social class are more likely to survive the disaster.
# Family members onboard (sibling, spouse, parents and children)
Overall, the pattern seems to be people with a few family member aboard has the highest chance of survival. Passengers with no family member or a lot of family aboard have a lower chance of survival.
The confidence intervals are relatively large at higher, suggesting the explanation power of those two various might be relatively weak.
Step4:  Age
Overall, The kernel density estimates look quite similar for those who survived and who didn't. This is the same for the boxplots. The median age for both groups are the same and the difference in quantiles are relatively small. 
One obvious misalignment of the two density plots happens when the passenger is a child ( < 14? ). The hump at far left of the the density curve of the suvived population suggests children had a higher chance of survival. 
One the flip side, young adults, is less likely to survive comparing to other age groups.
Logit Regression
Because our dependent variable (survived) is a binary variable 0 or 1. We use logit regression to study the influence of various factors on the probability of survival.
We will drop any data entries if it contains NA value in any of the features we are interested in. 
Also we will encode female to be 1 and male to be 0. Numercial representation is required for the regression to work.
<ASSISTANT_TASK:>
Python Code:
%pylab inline
import pandas as pd
import matplotlib.pylab as plt
import seaborn as sns
import statsmodels.api as sm
# read the data and inspect
titanic = pd.read_csv('titanic-data.csv')
print titanic.info()
titanic.head()
# drop those columns we are not interested in.
titanic.drop(["Name", "Ticket", "Cabin", "Embarked", "Fare"], axis=1, inplace=True)
Explanation: P2 - Titanic Data
<hr>
Questions
What are the most important factors related to the probability of survival ?
<hr>
Load Data
End of explanation
# helpers ----------------------------
def value_in_range(series, min, max): 
    assert pd.to_numeric(series, errors="raise").all() # make sure those are numbers
    return min <= series.min() and series.max() <= max
# ------------------------------------
# Sex can either be male or female
assert titanic["Sex"].isin(["male", "female"]).all() == True
# Survived 
assert titanic["Survived"].isin([0,1]).all() == True
# Pclass should be either 1, 2, or 3 and 
assert titanic["Pclass"].isin([1, 2, 3]).all() == True
# Age should be sensible, say between 0 to 100
AgeSeries = titanic[titanic["Age"].isnull() == False]["Age"]
assert value_in_range(AgeSeries, 0, 100) == True 
Explanation: Name and Embarked are dropped from the dataset because passenger name and embarking location shouldn't have any meaningful correlation with their chance of surviving. Arguably, the embarking locations might give some indication of passengers' social-economical background. However, I will be in favor of using "Pclass",  because it is specifically mentioned in the special notes in the data souce to be a proxy for social-economic status. 
I dropped Fare in favor of Pclass for the same reason. 
Cabin is an interesting feature. It is plausible the exact location of the passenger when the accident happens have an significant impact on the chance of survival. According to Wikipedia, the Titanic hits an iceburg at 11:40 pm. Given its already late in night, it is likely people will be sleeping at their cabin. However, there is only 204 data entries has a non-NaN value for this feature. Cabin is hence removed from the dataset as well.
Ticket number seems to be fairly random and not consistent across the dataset. It doesn't seems to contain any useful information. I have dropped from our dataset as well.
Other features like social-economic class (Pclass), gender (Sex), Age, and number of families onboard (Parch and SibSp) are all reasonable factors that could influence the chance of survival.
<hr />
Data Audit
End of explanation
# using seaborn's pointplot to visually explore the relationship between individual categorical varibles 
sns.set_style("whitegrid")
g = sns.PairGrid(titanic, x_vars=["Sex", "Pclass", "SibSp", "Parch"], y_vars=["Survived"], size=4)
g.map(sns.pointplot, ci=99)
g.axes[0,0].set_ylabel("survival rate")
g.fig.suptitle("Point Plots")
Explanation: <hr>
Data Exploration
Visualisation
We first use the pointplot to explore the effect of "Sex", "Pclass", "SibSp", and "Parch" on survival. The pointplot is great for categorical variable and numerical variable with small countable range. It provides the mean estimate and confidence interval for each possible value of the variable. 
For age, we will compare the kdeplot for both who survived and those didn't survive. If the kde are similar, then it indicates the age have little correlation with the chance of survival. If one kde value for survival is higher than the other at a particular age range, then it suggests the chance of survival is higher for than age group.
End of explanation
# ploting the kernal distribution for age 
figure = plt.figure()
ax_top = figure.add_subplot(211)
ax_top.set_xlim(0,85)
ax_top.set_xlabel("Age")
ax_top.set_ylabel("Proportional of Population")
ax_top.set_title("Kernal Density Estimate for Age grouped by survival")
ax_bottom = figure.add_subplot(212)
ax_bottom.set_xlim(0,85)
ax_bottom.set_title("Boxplot for Age distribution grouped by survival")
x = titanic[titanic["Survived"] == 1] 
y = titanic[titanic["Survived"] == 0]
_ = sns.kdeplot(x["Age"].dropna(),
                label="survived == True", 
                cut= True, shade=True, 
                ax=ax_top)
_ = sns.kdeplot(y["Age"].dropna(), 
                label="survived == False", 
                cut=True, shade=True, 
                ax=ax_top)
_ = sns.boxplot(x="Age",
                y="Survived",
                data=titanic.dropna(subset = ["Age"]),
                orient="h",
                ax=ax_bottom)
plt.tight_layout()
Explanation: Sex
The deep slope and small verticle bars (representing 99% confidence interval of mean) in the first visualisation suggests that "Sex" is strongly correlated with the chance of survival. Females have much higher chance of survival than male. 
Social class (Pclass)
Similar to "Sex", the second visualisation shows people with higher social class are more likely to survive the disaster.
# Family members onboard (sibling, spouse, parents and children)
Overall, the pattern seems to be people with a few family member aboard has the highest chance of survival. Passengers with no family member or a lot of family aboard have a lower chance of survival.
The confidence intervals are relatively large at higher, suggesting the explanation power of those two various might be relatively weak.
End of explanation
# Drop data points if contain NA in any feature.
titanic_dropna = titanic.dropna(subset=["Survived", "Age", "Sex", "Pclass", "SibSp", "Parch"])
# convert "Sex" to numberic representation, which is required for regressions.
titanic_dropna["Sex"] = titanic_dropna["Sex"].apply(lambda x: {"female": 0, "male": 1}[x])
dep = titanic_dropna["Survived"]
indep = titanic_dropna[["Sex", "Pclass", "SibSp", "Parch", "Age"]]
print(sm.Logit(dep, sm.add_constant(indep)).fit().get_margeff().summary())
Explanation: Age
Overall, The kernel density estimates look quite similar for those who survived and who didn't. This is the same for the boxplots. The median age for both groups are the same and the difference in quantiles are relatively small. 
One obvious misalignment of the two density plots happens when the passenger is a child ( < 14? ). The hump at far left of the the density curve of the suvived population suggests children had a higher chance of survival. 
One the flip side, young adults, is less likely to survive comparing to other age groups.
Logit Regression
Because our dependent variable (survived) is a binary variable 0 or 1. We use logit regression to study the influence of various factors on the probability of survival.
We will drop any data entries if it contains NA value in any of the features we are interested in. 
Also we will encode female to be 1 and male to be 0. Numercial representation is required for the regression to work.
End of explanation
<END_TASK>