<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 Analyzing the NYC Subway Dataset
Intro to Data Science
Step1:  Functions for Getting, Mapping, and Plotting Data
Step2:  Function for Basic Statistics
Step3:  Formulas Implemented
(i.e., not included in modules/packages)
Wendt's rank-biserial correlation $r$
$$r = 1 - \frac{2U}{n_{1}n_{2}}$$
Cohen's $d$ (and pooled standard deviation $s$)
$$d = \frac{\bar{x}{1} - \bar{x}{2}}{s}$$
$$s = \sqrt{\frac{(n_{1} - 1)s_{1}^{2} + (n_{2} - 1)s_{2}^{2}}{n_{1} + n_{2} - 2}}$$
Class for Mann-Whitney U Test
Step4:  Section 1. Statistical Test
<h3 id='1_1_a'>1.1.a Which statistical test did you use to analyse the NYC subway data?</h3>
The Mann-Whitney $U$ test was used to determine if there was a statistically significant difference between the number of reported entries on rainy and non-rainy occasions. This nonparametric test of the equality of two population medians from independent samples was used since the distribution of entries is non-normal (right-skewed) and their shape is the same, as seen visually via histograms, probability plots, and box plots, and as the result of the Shapiro-Wilk normality test (see <a href='IntroDS-ProjectOne-DataExploration-Supplement.ipynb#prep-for-stats' target='_blank'>Preparation for Statistical Tests</a>). However, since the sample sizes are so large, the parametric Welch's $t$-test likely could have been used (and, it was implemented for confirmation purposes, along with the nonparametric Wilcoxon signed-rank test; both agreed with the Mann-Whitney $U$ test results).
Testing Average Values of $p$, $r$, and $d$ for Various Sample Sizes
Step5:  As witnessed above, when rainy and non-rainy days from the data set are considered populations (as opposed to samples themselves), it takes significantly large sample sizes from each population (e.g., $n = 3000$, which is more than $30\%$ of the total number of rainy days in the data set) to attain low $p$-values<sup>1</sup> frequently enough to reject the null hypothesis of the Mann-Whitney $U$ test<sup>2</sup> with the critical values proposed below.
Moreover, using Wendt's rank-biserial correlation $r$ and Cohen's $d$ to measure effect size, the relatively low average value of $r$<sup>3</sup> and the low average value of $d$<sup>4</sup> both suggest that the difference between the two samples (and, thus, the two populations) is trivial, even though, according to the Mann-Whitney U test, the difference appears to be statistically signficant (and only then with extremely large samples)<sup>5</sup>. In other words, statistical significance $\neq$ practical significance.
Notes
<sup>1</sup> Identical samples would produce a large $p$ (e.g., $p \approx 0.49$); extremely different samples would produce a very small number (e.g., $p \approx 0$).
<sup>2</sup> Identical samples would produce $U = \frac{n^{2}}{2}$ (e.g., when $n = 450$, $U = 101250$); extremely different samples can produce a $U$ that is orders of magnitude smaller (e.g., when $n = 450$, possibly $U = 1293$).
<sup>3</sup> For very different samples, $r \rightarrow 1$; in the above tests, as $n$ increases, $r \rightarrow 0$.  
<sup>4</sup> For very different samples, $d \rightarrow 1$; in the above tests, as $n$ increases, $d$ tends to remain constant, $d \approx 0.06$, even when the sample size is extremely large. $d$ is interpreted as the difference in the number of standard deviations.
<sup>5</sup> On the issue of $p$-values and large data sets, see Lin, M., Lucas, H.C., and Shmueli, G. Research Commentary—Too big to fail
Step6:  The Mann-Whitney $U$ test is a nonparametric test of the null hypothesis that the distributions of two populations are the same.
To verify the assumption that the simple, randomly sampled values are independent, the sample sizes should be less than $5\%$ of the population sizes ($n \lt 0.05N$). Since the maximum number of rainy days is $9585$ ($N = 9585$), a reasonable sample size for each group would be $450$ ($n = 450$). 
Null Hyptohesis
$H_{0}$
<ASSISTANT_TASK:>
Python Code:
import inflect # for string manipulation
import numpy as np
import pandas as pd
import scipy as sp
import scipy.stats as st
import matplotlib.pyplot as plt
%matplotlib inline
filename = '/Users/excalibur/py/nanodegree/intro_ds/final_project/improved-dataset/turnstile_weather_v2.csv'
# import data
data = pd.read_csv(filename)
Explanation: Analyzing the NYC Subway Dataset
Intro to Data Science: Final Project 1, Part 2
(Short Questions)
Section 1. Statistical Test
Austin J. Alexander
Import Directives and Initial DataFrame Creation
End of explanation
entries_hourly_by_row = data['ENTRIESn_hourly'].values
def map_column_to_entries_hourly(column):
    instances = column.values # e.g., longitude_instances = data['longitude'].values
    
    # reduce
    entries_hourly = {} # e.g., longitude_entries_hourly = {}
    for i in np.arange(len(instances)): 
        if instances[i] in entries_hourly:
            entries_hourly[instances[i]] += float(entries_hourly_by_row[i])
        else:
            entries_hourly[instances[i]] = float(entries_hourly_by_row[i])
            
    return entries_hourly # e.g., longitudes, entries
def create_df(entries_hourly_dict, column1name):
    # e.g, longitude_df = pd.DataFrame(data=longitude_entries_hourly.items(), columns=['longitude','entries'])
    df = pd.DataFrame(data=entries_hourly_dict.items(), columns=[column1name,'entries'])
    
    return df # e.g, longitude_df
rain_entries_hourly = map_column_to_entries_hourly(data['rain'])
rain_df = create_df(rain_entries_hourly, 'rain')
rain_days = data[data['rain'] == 1]
no_rain_days = data[data['rain'] == 0]
def plot_box(sample1, sample2):
    plt.boxplot([sample2, sample1], vert=False)
    plt.title('NUMBER OF ENTRIES PER SAMPLE')
    plt.xlabel('ENTRIESn_hourly')
    plt.yticks([1, 2], ['Sample 2', 'Sample 1'])
    plt.show()
Explanation: Functions for Getting, Mapping, and Plotting Data
End of explanation
def describe_samples(sample1, sample2):
    size1, min_max1, mean1, var1, skew1, kurt1 = st.describe(sample1)
    size2, min_max2, mean2, var2, skew2, kurt2 = st.describe(sample2)
    med1 = np.median(sample1)
    med2 = np.median(sample2)
    std1 = np.std(sample1)
    std2 = np.std(sample2)
    print "Sample 1 (rainy days):\n  min = {0}, max = {1},\n  mean = {2:.2f}, median = {3}, var = {4:.2f}, std = {5:.2f}".format(min_max1[0], min_max1[1], mean1, med1, var1, std1)
    print "Sample 2 (non-rainy days):\n  min = {0}, max = {1},\n  mean = {2:.2f}, median = {3}, var = {4:.2f}, std = {5:.2f}".format(min_max2[0], min_max2[1], mean2, med2, var2, std2)
Explanation: Function for Basic Statistics
End of explanation
class MannWhitneyU:
    
    def __init__(self,n):
        self.n = n
        self.num_of_tests = 1000
        self.sample1 = 0
        self.sample2 = 0
        
    def sample_and_test(self, plot, describe):
        self.sample1 = np.random.choice(rain_days['ENTRIESn_hourly'], size=self.n, replace=False)
        self.sample2 = np.random.choice(no_rain_days['ENTRIESn_hourly'], size=self.n, replace=False)
        ### the following two self.sample2 assignments are for testing purposes ###
        #self.sample2 = self.sample1 # test when samples are same
        #self.sample2 = np.random.choice(np.random.randn(self.n),self.n) # test for when samples are very different 
        
        if plot == True:
            plot_box(self.sample1,self.sample2)
        
        if describe == True:
            describe_samples(self.sample1,self.sample2)
        
        return st.mannwhitneyu(self.sample1, self.sample2)
    
    def effect_sizes(self, U):
        # Wendt's rank-biserial correlation
        r = (1 - np.true_divide((2*U),(self.n*self.n)))
        # Cohen's d
        s = np.sqrt(np.true_divide((((self.n-1)*np.std(self.sample1)**2) + ((self.n-1)*np.std(self.sample2)**2)), (self.n+self.n-2)))
        d = np.true_divide((np.mean(self.sample1) - np.mean(self.sample2)), s)
        return r,d
    def trial_series(self):
        success = 0
        U_values = []
        p_values = []
        d_values = []
        r_values = []
        for i in np.arange(self.num_of_tests):
            U, p = self.sample_and_test(False, False)
            r, d = self.effect_sizes(U)
            U_values.append(U)
            # scipy.stats.mannwhitneyu returns p for a one-sided hypothesis, 
            # so multiply by 2 for two-sided
            p_values.append(p*2)
            d_values.append(d)
            r_values.append(r)
            if p <= 0.05:
                success += 1
        print "n = {0}".format(self.n)
        print "average U value: {0:.2f}".format(np.mean(U_values))
        print "number of times p <= 0.05: {0}/{1} ({2}%)".format(success, self.num_of_tests, (np.true_divide(success,self.num_of_tests)*100))
        print "average p value: {0:.2f}".format(np.mean(p_values))
        print "average rank-biserial r value: {0:.2f}".format(np.mean(r_values))
        print "average Cohen's d value: {0:.2f}".format(np.mean(d_values))
        
        plt.hist(p_values, color='green', alpha=0.3)
        plt.show()
Explanation: Formulas Implemented
(i.e., not included in modules/packages)
Wendt's rank-biserial correlation $r$
$$r = 1 - \frac{2U}{n_{1}n_{2}}$$
Cohen's $d$ (and pooled standard deviation $s$)
$$d = \frac{\bar{x}{1} - \bar{x}{2}}{s}$$
$$s = \sqrt{\frac{(n_{1} - 1)s_{1}^{2} + (n_{2} - 1)s_{2}^{2}}{n_{1} + n_{2} - 2}}$$
Class for Mann-Whitney U Test
End of explanation
sample_sizes = [30, 100, 500, 1500, 3000, 5000, 9585]
for n in sample_sizes:
    MannWhitneyU(n).trial_series()
Explanation: Section 1. Statistical Test
<h3 id='1_1_a'>1.1.a Which statistical test did you use to analyse the NYC subway data?</h3>
The Mann-Whitney $U$ test was used to determine if there was a statistically significant difference between the number of reported entries on rainy and non-rainy occasions. This nonparametric test of the equality of two population medians from independent samples was used since the distribution of entries is non-normal (right-skewed) and their shape is the same, as seen visually via histograms, probability plots, and box plots, and as the result of the Shapiro-Wilk normality test (see <a href='IntroDS-ProjectOne-DataExploration-Supplement.ipynb#prep-for-stats' target='_blank'>Preparation for Statistical Tests</a>). However, since the sample sizes are so large, the parametric Welch's $t$-test likely could have been used (and, it was implemented for confirmation purposes, along with the nonparametric Wilcoxon signed-rank test; both agreed with the Mann-Whitney $U$ test results).
Testing Average Values of $p$, $r$, and $d$ for Various Sample Sizes
End of explanation
print "Shape of rainy-days data:" +str(rain_days.shape)
N = rain_days.shape[0]
print "N = " + str(N)
print "0.05 * N = " + str(0.05 * N)
Explanation: As witnessed above, when rainy and non-rainy days from the data set are considered populations (as opposed to samples themselves), it takes significantly large sample sizes from each population (e.g., $n = 3000$, which is more than $30\%$ of the total number of rainy days in the data set) to attain low $p$-values<sup>1</sup> frequently enough to reject the null hypothesis of the Mann-Whitney $U$ test<sup>2</sup> with the critical values proposed below.
Moreover, using Wendt's rank-biserial correlation $r$ and Cohen's $d$ to measure effect size, the relatively low average value of $r$<sup>3</sup> and the low average value of $d$<sup>4</sup> both suggest that the difference between the two samples (and, thus, the two populations) is trivial, even though, according to the Mann-Whitney U test, the difference appears to be statistically signficant (and only then with extremely large samples)<sup>5</sup>. In other words, statistical significance $\neq$ practical significance.
Notes
<sup>1</sup> Identical samples would produce a large $p$ (e.g., $p \approx 0.49$); extremely different samples would produce a very small number (e.g., $p \approx 0$).
<sup>2</sup> Identical samples would produce $U = \frac{n^{2}}{2}$ (e.g., when $n = 450$, $U = 101250$); extremely different samples can produce a $U$ that is orders of magnitude smaller (e.g., when $n = 450$, possibly $U = 1293$).
<sup>3</sup> For very different samples, $r \rightarrow 1$; in the above tests, as $n$ increases, $r \rightarrow 0$.  
<sup>4</sup> For very different samples, $d \rightarrow 1$; in the above tests, as $n$ increases, $d$ tends to remain constant, $d \approx 0.06$, even when the sample size is extremely large. $d$ is interpreted as the difference in the number of standard deviations.
<sup>5</sup> On the issue of $p$-values and large data sets, see Lin, M., Lucas, H.C., and Shmueli, G. Research Commentary—Too big to fail: Large samples and the P-value problem. Inf. Syst. Res. 2013; 24: 906–917. PDF <a href='http://www.galitshmueli.com/system/files/Print%20Version.pdf' target='_blank'>here</a>.
<h3 id='1_1_b'>1.1.b Did you use a one-tail or a two-tail P value?</h3>
A two-tail $p$-value was selected since an appropriate initial question, given the results of the <a href='IntroDS-ProjectOne-DataExploration-Supplement.ipynb#weather-related' target='_blank'>Weather-Related Data</a> section of the DataExploration supplement, is simply whether or not there is a statistically significant difference between the populations (i.e., not whether one population is statistically-significantly greater than another).
<h3 id='1_1_c'>1.1.c What is the null hypothesis?</h3>
End of explanation
n = 450
mwu = MannWhitneyU(n)
U, p = mwu.sample_and_test(True,True)
r, d = mwu.effect_sizes(U)
print "\nMann-Whitney U test results:"
print "n = {0}".format(n)
print "U = {0}".format(U)
print "p = {0:.2f}".format(np.mean(p))
print "rank-biserial r value: {0:.2f}".format(np.mean(r))
print "Cohen's d value: {0:.2f}".format(np.mean(d))
Explanation: The Mann-Whitney $U$ test is a nonparametric test of the null hypothesis that the distributions of two populations are the same.
To verify the assumption that the simple, randomly sampled values are independent, the sample sizes should be less than $5\%$ of the population sizes ($n \lt 0.05N$). Since the maximum number of rainy days is $9585$ ($N = 9585$), a reasonable sample size for each group would be $450$ ($n = 450$). 
Null Hyptohesis
$H_{0}$: $M_{1} = M_{2}$ or $H_{0}$: $M_{1} - M_{2} = 0$
Alternate Hypothesis 
$H_{1}$: $M_{1} \neq M_{2}$ or $H_{1}$: $M_{1} - M_{2} \neq 0$
A $95\%$ level of confidence would suggest that $95\%$ of samples would produce similar statistical results.
For a $95\%$ level of confidence, the level of significance (i.e., the probability of making a Type I error)  $\alpha = (1 - 0.05) \cdot 100\% = 0.05$.
<h3 id='1_1_d'>1.1.d What is your p-critical value?</h3>
$p \leq 0.05$
Gather New Samples and Perform Statistical Test
End of explanation
<END_TASK>