<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 Titanic Data Analysis By Gangadhara Naga Sai
Overview Of Titanic Dataset
In 1912, the ship RMS Titanic struck an iceberg on its maiden voyage and sank, resulting in the deaths of most of its    passengers and crew. In this project, we will explore the RMS Titanic passenger manifest to determine whether someone survived or did not survive.Demographics and passenger information from 891 of the 2224 passengers and crew on board the Titanic Dataset is obtained from kaggle (https
Step1:  Data Description
From a sample of the RMS Titanic data, we can see the various features present for each passenger on the ship
Step2:  From the above info(),We can see columns Age, Cabin and Embarked have missing values.
Handling the missing values
Step3:  lets decide should we remove missing age rows or fill the missing values with the mean, I'm going first to split the sample data into 2 samples  with missing age and with age and perform a t test
Step4:  Hypothesis
To fill the missing data with mean, i will decide with t test by being sure that passengers in these 2 samples are likely to have the similar survival rate.
H0
Step5:  p value is than 0.05 which results in rejecting H0 ,so there is a significant difference in mean .So, I'm going to substitute the missing values with the mean age.
Step6:  We can see that all columns have identical length.
Data Exploration and Visualization
Step7:  From the above statistics
- Youngest to survive
Step8:  These must be new borns and all survived
Step9:  From the above plot we can see that gender played an important role in survival of each
individaul
Female Survival rate 
Step10:  From the above plot  we can clearly see individuals of different class distibuted for various ages. And the red line shows the average of age for each class
Step11:  A passenger from Class 1 is about 2.5x times more likely to survive than a passenger in Class 3. 
Social-economic standing was a factor in survival rate of passengers.
Class 1
Step12:  From the above plot we can see how female individuals are given 1st preference and based on class.
Social-economic standing was a factor in survival rate of passengers by gender
Class 1 - female survival rate
Step13:  From the above boxplot and calculated mean
Step14:  These are percentage of survivors for Group_age
- Adult 
Step15:  Women and children are given importance in the survival of a number of people. 
Man 16.57%
Women 75.2%
Child 54%
<ASSISTANT_TASK:>
Python Code:
import numpy as np
import pandas as pd
from IPython.display import display
%matplotlib inline
# Load the dataset
files = 'titanic_data.csv'
data_titanic = pd.read_csv(files)
display(data_titanic.head())
Explanation: Titanic Data Analysis By Gangadhara Naga Sai
Overview Of Titanic Dataset
In 1912, the ship RMS Titanic struck an iceberg on its maiden voyage and sank, resulting in the deaths of most of its    passengers and crew. In this project, we will explore the RMS Titanic passenger manifest to determine whether someone survived or did not survive.Demographics and passenger information from 891 of the 2224 passengers and crew on board the Titanic Dataset is obtained from kaggle (https://www.kaggle.com/c/titanic/data).
Questions
Depending upon weather the passengers survived or did not survive based on the classification of
Age
Gender
Passenger class
Shows difference in survival chances of Each indiviudal?
Data Wrangling
End of explanation
data =data_titanic
# Show the dataset 
display(data.head())
data.info()
Explanation: Data Description
From a sample of the RMS Titanic data, we can see the various features present for each passenger on the ship:
- Survived: Outcome of survival (0 = No; 1 = Yes)
- Pclass: Socio-economic class (1 = Upper class; 2 = Middle class; 3 = Lower class)
- Name: Name of passenger
- Sex: Sex of the passenger
- Age: Age of the passenger (Some entries contain NaN)
- SibSp: Number of siblings and spouses of the passenger aboard
- Parch: Number of parents and children of the passenger aboard
- Ticket: Ticket number of the passenger
- Fare: Fare paid by the passenger
- Cabin Cabin number of the passenger (Some entries contain NaN)
- Embarked: Port of embarkation of the passenger (C = Cherbourg; Q = Queenstown; S = Southampton)
Variable Notes
pclass: A proxy for socio-economic status (SES)
- 1st = Upper
- 2nd = Middle
- 3rd = Lower
age: Age is fractional if less than 1. If the age is estimated, is it in the form of xx.5
sibsp: The dataset defines family relations in this way...
Sibling = brother, sister, stepbrother, stepsister
Spouse = husband, wife (mistresses and fiancés were ignored)
parch: The dataset defines family relations in this way...
Parent = mother, father
Child = daughter, son, stepdaughter, stepson
Some children travelled only with a nanny, therefore parch=0 for them.
End of explanation
#exculding some coloumns
del data['Ticket']
del data['Cabin']
del data['Embarked']
del data['Name']
del data['PassengerId']
del data['Fare']
data.describe(include='all')
# Calculate number of missing values
data.isnull().sum()
null_female = data[pd.isnull(data['Age'])]['Sex'] == 'female'
null_male = data[pd.isnull(data['Age'])]['Sex'] == 'male'
print "Total missing age for female:",null_female.sum()
print "Total missing age for male:",null_male.sum()
Explanation: From the above info(),We can see columns Age, Cabin and Embarked have missing values.
Handling the missing values:
Ignore the rows with missing data,
Exclude the variable at all or we might substite it with  mean or median.
Age  80% of the data is available,which seems a important variable so not to exclude.
Port of embarkation doesn't seem interesting.
cabin 23% of the data so decided to exclude.
PassengerId,Name,fare doesnt seem to contribute to any survival investigation
End of explanation
notnull_age = data[pd.notnull(data['Age'])]
null_age = data[pd.isnull(data['Age'])]
Explanation: lets decide should we remove missing age rows or fill the missing values with the mean, I'm going first to split the sample data into 2 samples  with missing age and with age and perform a t test
End of explanation
from scipy.stats import ttest_ind
ttest_ind(notnull_age['Survived'], null_age['Survived'])
Explanation: Hypothesis
To fill the missing data with mean, i will decide with t test by being sure that passengers in these 2 samples are likely to have the similar survival rate.
H0:means of the samples populatins are equal 
H1: population means are different
If the resulted p value is going to be less than the critical value (with alpha level 0.05),
I should reject the null hypothesis and conclude that population means are different not by chance
(Ignoring the data of missing data which almost 20% of the data should be neglected).
I'm using the existing in scipy.stats function to perform t test for independent variables:
End of explanation
print "Age median values by Age and Sex:"
#we are grouping by gender and class and taking median of age so we can replace with corrresponding values instead of NaN
print data.groupby(['Sex','Pclass'], as_index=False).median().loc[:, ['Sex','Pclass', 'Age']]
print "Age values for 5 first persons in dataset:"
print data.loc[data['Age'].isnull(),['Age','Sex','Pclass']].head(5)
# apply transformation: Age missing values are filled with regard to Pclass and Sex:
data.loc[:, 'Age'] = data.groupby(['Sex','Pclass']).transform(lambda x: x.fillna(x.median()))
print data.loc[[5,17,19,26,28],['Age','Sex','Pclass']].head(5)
data['Age'] = data['Age'].fillna(data['Age'].mean())
data.describe(include='all')
Explanation: p value is than 0.05 which results in rejecting H0 ,so there is a significant difference in mean .So, I'm going to substitute the missing values with the mean age.
End of explanation
data_s=data
survival_group = data_s.groupby('Survived')
survival_group.describe()
Explanation: We can see that all columns have identical length.
Data Exploration and Visualization
End of explanation
#  Seriously i dont understand why age is 0.42
data_s[data_s['Age'] < 1]
Explanation: From the above statistics
- Youngest to survive: 0.42 
- Youngest to die: 1.0 
- Oldest to survive: 80.0 
- Oldest to die: 74.0
End of explanation
import matplotlib.pyplot as plt
import seaborn as sns
# Set style for all graphs
#sns.set_style("light")
#sns.set_style("whitegrid")
sns.set_style("ticks", {"xtick.major.size": 8, "ytick.major.size": 8})
def plot(a,i):
    fig=plt.figure() #Plots in matplotlib reside within a figure object, use plt.figure to create new figure
    #Create one or more subplots using add_subplot, because you can't create blank figure
    ax = fig.add_subplot(1,1,1)
    #Variable
    ax.hist(data[a],bins = i) # Here you can play with number of bins
    plt.title(a + ' distribution')
    plt.xlabel(a)
    plt.ylabel('Passenger Count')
    plt.show()
plot("Age",30)
print "The above distribution of Age seems a little bit deviating from normal distribution"
print 
plot("SibSp",8)
print "The above distribution of SibSp seems a right-skewed distribution"
plot("Parch",6)
print "The above distribution of Age seems a right-skewed distribution"
sns.factorplot(x="Sex", y="Age", data=data_s, kind="box", size=7, aspect=.8)\
.set_xticklabels(["Male","Female"])
plt.title('Boxplot of Age grouped by sex')
print "From the below plot we can see there are more elderly men than women and average age for men is higher than women"
Explanation: These must be new borns and all survived
End of explanation
sns.factorplot(x="Pclass", y="Age", data=data_s, kind="box", size=7, aspect=.8)\
.set_xticklabels(["1","2","3"])
plt.title('Boxplot of Age grouped by sex')
print "From the below plot we can see the average age is decreasing from calss 1 to class 3"
Explanation: From the above plot we can see that gender played an important role in survival of each
individaul
Female Survival rate : 74.2%
Male Survival rate: 18.8%
End of explanation
 sns.factorplot( 'Sex' , 'Survived', data = data, kind = 'bar')
plt.title('Histogram of Survival rate grouped by Sex')
print "From the plot we can clearly see the survival rate of female is very high"
## GENDER
survivals = pd.crosstab([ data_s.Sex], data_s.Survived.astype(bool))
survivals.plot(kind='bar', stacked=False)
plt.ylabel("Passenger count")
plt.title('Histogram of Passenger count grouped by Sex and survived')
survival = data_s.groupby('Sex')['Survived']
survival.mean()
#PCLASS
survivals = pd.crosstab([data_s.Pclass], data_s.Survived.astype(bool))
survivals.plot(kind='bar', stacked=True)
plt.ylabel("Passenger count")
plt.title('Histogram of Passenger count grouped by Class')
survival=data.groupby(['Pclass'])
survival.mean()
Explanation: From the above plot  we can clearly see individuals of different class distibuted for various ages. And the red line shows the average of age for each class
End of explanation
survivals = pd.crosstab([data_s.Pclass, data_s.Sex], data_s.Survived.astype(bool))
survivals.plot(kind='bar', stacked=True)
survive=data.groupby(['Sex','Pclass'])
plt.ylabel("Passenger count")
plt.title('Histogram of passenger count grouped by sex and Class')
#survive.Survived.sum().plot(kind='barh')
survive.mean()
Explanation: A passenger from Class 1 is about 2.5x times more likely to survive than a passenger in Class 3. 
Social-economic standing was a factor in survival rate of passengers.
Class 1: 62.96%
Class 2: 47.28%
Class 3: 24.24%
End of explanation
#Age
sns.factorplot(x="Survived", y="Age", hue='Sex', data=data_s, kind="box", size=7, aspect=.8)\
.set_xticklabels(["Expired","Survived"])
plt.title('Boxplot of Age grouped by sex and Survival')
# survive_A=data.groupby(['Sex','Age'])
Explanation: From the above plot we can see how female individuals are given 1st preference and based on class.
Social-economic standing was a factor in survival rate of passengers by gender
Class 1 - female survival rate: 96.81%
Class 1 - male survival rate: 36.89%
Class 2 - female survival rate: 92.11%
Class 2 - male survival rate: 15.74%
Class 3 - female survival rate: 50.0%
Class 3 - male survival rate: 13.54%
End of explanation
#Age
# We are dividing the Age data into 3 buckets of (0-18),(18-40),(40-90)
# and labeling them as 'Childs','Adults','Seniors' respectively
data['group_age']  = pd.cut(data['Age'], bins=[0,18,40,90], labels=['Childs','Adults','Seniors'])
data.head(5)
    
survive_a=data.groupby(['group_age'])
survival_a = pd.crosstab([data.group_age], data_s.Survived.astype(bool))
survival_a.plot(kind='bar', stacked=True)
plt.title('Bar plot of Passenger count grouped by age categories ')
plt.ylabel("Passenger count")
# sns.factorplot(x="group_age", y="Age", hue='Sex', data=data, kind="box", size=7, aspect=.8)
survive_a.mean()
Explanation: From the above boxplot and calculated mean:
- Irrespective of sex and class, age was not a deciding factor in the passenger survival rate
- Average age for surived and not survived seemed almost same from the boxplot
End of explanation
def group(d,v):
    if (d == 'female') and (v >= 18):
        return 'Woman'
    elif v < 18:
        return 'child'                        
    elif (d == 'male') and (v >= 18): 
        return  'Man'
data['Category'] = data.apply(lambda row:group(row['Sex'], row['Age']), axis=1) 
data.head(5)
survival_a = pd.crosstab([data.Category], data_s.Survived.astype(bool))
survival_a.plot(kind='bar', stacked=True)
plt.ylabel("Passenger count")
plt.title('Survival by Age category')
data.groupby(['Category']).mean()["Survived"]
Explanation: These are percentage of survivors for Group_age
- Adult : 36.0%
- Child : 50.35%
- Senior: 36.66%
Women and children have preference First to lifeboats?
End of explanation
g = sns.factorplot(x="Category", y="Survived", col="Pclass", data=data, 
                   saturation=.5, kind="bar", ci=None, size=5, aspect=.8)
# Fix up the labels
(g.set_axis_labels('', 'Survival Rate')
     .set_xticklabels(["Men", "Women","child"])
     .set_titles("Class {col_name}")
     .set(ylim=(0, 1))
     .despine(left=True, bottom=True))
print 'Histogram of Survival rate grouped by Age Category and Class:'
# We are dividing the Age data into 3 buckets of (0-18),(18-40),(40-90)
# and labeling them as 'Childs','Adults','Seniors' respectively
data['group_age']  = pd.cut(data['Age'], bins=[0,18,40,90], labels=['Childs','Adults','Seniors'])
#finding mean Survival rate grouped by 'group_age','Sex'.
df = data.groupby(['group_age','Sex'],as_index=False).mean().loc[:,['group_age','Sex','Survived']]
f, (ax1, ax2,ax3) = plt.subplots(1, 3,figsize=(15,7))
g = sns.barplot(x="group_age", y="Survived", hue="Sex", data=df,ax=ax1)
ax1.set_title('Survival by Age and Sex')
#finding mean Survival rate grouped by 'group_age'.
data2 = data.groupby(['group_age'],as_index=False).mean().loc[:,['group_age','Survived']]
h = sns.barplot(x="group_age",y='Survived', data=data2,ax=ax2)
ax2.set_title('Survival by Age')
#finding mean Survival rate grouped by 'group_age'.
data3 = data.groupby(['group_age'],as_index=False).count().loc[:,['group_age','Survived']]
hh = sns.barplot(x="group_age",y='Survived', data=data3,ax=ax3)
ax3.set_title('Age distribution in Ship')
ax3.set_ylabel('Age Distribution')
for ax in f.axes:
    plt.sca(ax)
    plt.xticks(rotation=90)
plt.tight_layout()
plt.show()
data_C=data.groupby(['Category',"Pclass"]).mean()
data_C.sort("Survived")["Survived"]
Explanation: Women and children are given importance in the survival of a number of people. 
Man 16.57%
Women 75.2%
Child 54%
End of explanation
<END_TASK>