<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 Statistics 2
Step1:  Cleaning and filling data
Checking to see what columns need to be filled using the .info() method.
Step2:  Filling all NaN ages with the mean of all the ages and confirming with .info() method. We later compensate for this with a functions that remove the mean.  This will be pointed out with a 'COMPENSATION
Step3:  Question 1
Calculate the probability of survival during Titanic crash.
Step4:  Question 2
CHOOSE TWO OF...
A passenger was male
A passenger was female and had atleast 1 SibSp on board
A survivor was from Cherbourg
Our first choice of question is to see the probability that a passenger was male.
Step5:  Our second choice of question is to find the probablity that a survivor was from Cherbourg.
Step6:  Question 3
Plot the distribution of passenger ages. (Bins = 25)
COMPENSATION
Step7:  Question 4
Find the probability that a passenger was less than 10 years old.
Step8:  Question 5
Given 100 passengers at random, determine the probability that exactly 42 passengers survive.
Step9:  There is a 0.0613 probability that exactly 42 passenger survive out of 100.  See above 'Out' for a more precise probability.
Question 5
What is the probability that at least 42 of those 100 passenger survive?
Step10:  There is a 0.198 probability that at least 42 of those 100 passenger survive.  See above 'Out' for a more precise probability.
Question 6
Is there a statistically significant difference between the age of male and female survivors?
COMPENSATION
Step11:  There is no significance between the age of female and male survivors. This is because the p-value is greater than 0.05.
COMPENSATION
Step12:  Question 7
Is there a statistically significant difference between the fares paid by passengers between Queentown and the passengers from Cherbourg?
Step13:  There is statistical difference in the fares paid between the passengers at Queentown and Cherboug. This is indicated by the p-value that is less than 0.01.
<ASSISTANT_TASK:>
Python Code:
import numpy as np
import pandas as pd
titanic_data = pd.read_csv('train.csv')
titanic_data.head(5)
Explanation: Statistics 2: Probabilities, Distributions, and Tests
Notebook Summary
This notebook is a series of exercises to practice utilizing probabiliteis, distributions and tests by answering questions in relation to titianic passenger data.  The following will be presented with headers that incorperate the questions followed by the calculations and a written summary of the result.
Importing data file
End of explanation
titanic_data.info()
Explanation: Cleaning and filling data
Checking to see what columns need to be filled using the .info() method.
End of explanation
titanic_data.Age = titanic_data.Age.fillna(np.mean(titanic_data.Age))
titanic_data.info()
Explanation: Filling all NaN ages with the mean of all the ages and confirming with .info() method. We later compensate for this with a functions that remove the mean.  This will be pointed out with a 'COMPENSATION:' and define the action as it arises.
End of explanation
survivors = titanic_data[titanic_data.Survived == 1]
survivor_prob = (len(survivors) / len(titanic_data))
print("There is a " + str(survivor_prob) + " percent chance of survival.")
Explanation: Question 1
Calculate the probability of survival during Titanic crash.
End of explanation
male_passenger = titanic_data[titanic_data.Sex == 'male']
prob_male = (len(male_passenger) / len(titanic_data))
print("There is a " + str(prob_male) + " percent probability that a passenger was male.")
Explanation: Question 2
CHOOSE TWO OF...
A passenger was male
A passenger was female and had atleast 1 SibSp on board
A survivor was from Cherbourg
Our first choice of question is to see the probability that a passenger was male.
End of explanation
c_port = survivors[survivors.Embarked == 'C']
prob_c = (len(c_port) / len(survivors))
print("There is a " + str(prob_c) + " percent probability that a survivor was from Cherbourg.")
Explanation: Our second choice of question is to find the probablity that a survivor was from Cherbourg.
End of explanation
import matplotlib.pyplot as plt
%matplotlib inline
all_ages = []
age_mean = np.mean(titanic_data.Age)
for i, k in enumerate(titanic_data.Age): 
    if round(k, 3) != round(age_mean, 3):
        all_ages.append(k)
H, edges = np.histogram(all_ages, bins=25)
ax = plt.subplot(111)
ax.bar(edges[:-1], H / float(sum(H)), width=edges[1] - edges[0])
ax.set_xlabel("Passenger Age")
ax.set_ylabel("Frequency of Being on Board")
ax.minorticks_on()
plt.show()
Explanation: Question 3
Plot the distribution of passenger ages. (Bins = 25)
COMPENSATION: Earlier all the ages that were replaced with the mean.  Because that would centrally skew the distribution histogram we round the passenger age decimial to the third and match it against the mean roudned to the third decimal and remove that value from the data set.  By removing those fills the data will be represented correctly.   (This is waht the for loop does below.)
End of explanation
less_then_ten = []
for i in all_ages:
    if i < 10:
        less_then_ten.append(i)
        
prob_less_then_ten = (len(less_then_ten) / len(all_ages))
print("There is a " + str(round(prob_less_then_ten, 3)) + " probabililty that a passenger was less then 10 year old.") 
Explanation: Question 4
Find the probability that a passenger was less than 10 years old.
End of explanation
from scipy.stats import binom
binom.pmf(42, 100, survivor_prob)
Explanation: Question 5
Given 100 passengers at random, determine the probability that exactly 42 passengers survive.
End of explanation
1 - binom.cdf(42, 100, survivor_prob)
Explanation: There is a 0.0613 probability that exactly 42 passenger survive out of 100.  See above 'Out' for a more precise probability.
Question 5
What is the probability that at least 42 of those 100 passenger survive?
End of explanation
from scipy.stats import ttest_ind
survivors_male = survivors[(survivors.Sex == 'male') & (round(survivors.Age,3) != round(age_mean, 3)) ]
survivors_female = survivors[(survivors.Sex == 'female') & (round(survivors.Age, 3) != round(age_mean, 3))]
t_stat, p_value = ttest_ind(survivors_male.Age, survivors_female.Age)
print("Results:\n\tt-statistic: %.5f\n\tp-value: %.5f" % (t_stat, p_value))
Explanation: There is a 0.198 probability that at least 42 of those 100 passenger survive.  See above 'Out' for a more precise probability.
Question 6
Is there a statistically significant difference between the age of male and female survivors?
COMPENSATION:  Within our male and female survival groups we also had a match to throw out any age that matches the mean to the third the decimal to remove the skew.  Post skew compensation increased the p value, indicating a correct compensation.
End of explanation
plt.figure(figsize=(10, 4))
opacity = 0.5
plt.hist(survivors_male.Age, bins=np.arange(0, 90, 5), alpha=opacity, label="Males")
plt.hist(survivors_female.Age, bins=np.arange(0, 90, 5), alpha=opacity, label="Females")
plt.legend()
plt.title("Age Distribution of Female and Male Survivors")
plt.xlabel("Ages")
plt.ylabel("Survival")
plt.show()
Explanation: There is no significance between the age of female and male survivors. This is because the p-value is greater than 0.05.
COMPENSATION: The age ditribution plot below against survivors also has our compensation affect which reduced a spike of survivors at the 25-30 age range.
End of explanation
from scipy.stats import ttest_ind
fare_from_q = titanic_data[titanic_data.Embarked == 'Q']
fare_from_c = titanic_data[titanic_data.Embarked == 'C']
t_stat, p_value = ttest_ind(fare_from_q.Fare, fare_from_c.Fare)
print("Results:\n\tt-statistic: %.5f\n\tp-value: %g" % (t_stat, p_value))
Explanation: Question 7
Is there a statistically significant difference between the fares paid by passengers between Queentown and the passengers from Cherbourg?
End of explanation
plt.figure(figsize=(10, 4))
opacity = 0.5
plt.hist(fare_from_q.Fare, bins=np.arange(0, 90, 5), alpha=opacity, label="Queenstown")
plt.hist(fare_from_c.Fare, bins=np.arange(0, 90, 5), alpha=opacity, label="Cherbourg")
plt.legend()
plt.title("Fare Distribution from Queenstown to Cherbourg")
plt.xlabel("Fare Price")
plt.ylabel("Number of Passengers")
plt.show()
Explanation: There is statistical difference in the fares paid between the passengers at Queentown and Cherboug. This is indicated by the p-value that is less than 0.01.
End of explanation
<END_TASK>