<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 Project 2
In this project, you will implement the exploratory analysis plan developed in Project 1. This will lay the groundwork for our our first modeling exercise in Project 3.
Step 1
Step1:  Step 2
Step2:  Questions
Question 1. How many observations are in our dataset?
Step3:  Answer
Step4:  Question 3. Why would GRE have a larger STD than GPA?
Answer
Step5:  Question 5. Confirm that you dropped the correct data. How can you tell?
Answer
Step6:  Question 7. What do this plots show?
Answer
Step7:  Question 9.  If our model had an assumption of a normal distribution would we meet that requirement?
Answer
Step8:  Question 12. What did you find?
Answer
<ASSISTANT_TASK:>
Python Code:
#imports
from __future__ import division
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import statsmodels.api as sm
import pylab as pl
import numpy as np
%matplotlib inline
Explanation: Project 2
In this project, you will implement the exploratory analysis plan developed in Project 1. This will lay the groundwork for our our first modeling exercise in Project 3.
Step 1: Load the python libraries you will need for this project
End of explanation
#Read in data from source 
df_raw = pd.read_csv("../assets/admissions.csv")
print df_raw.head()
Explanation: Step 2: Read in your data set
End of explanation
df_raw['admit'].count()
df_raw['gpa'].count()
df_raw.shape
rows,columns = df_raw.shape
print(rows)
print(columns)
Explanation: Questions
Question 1. How many observations are in our dataset?
End of explanation
#function
def summary_table():
    #creates a summary table for df_raw using .describe()
    df_raw.describe()
    return x
    print X
df_raw.describe()
Explanation: Answer: 400 observations. These 400 observations are displayed within 4 rows of 400 observations each.
Question 2. Create a summary table
End of explanation
df_raw.dropna() 
#drops any missing data rows from admissions.csv dataset 
#returns 397 observations (complete observation rows) across 4 columns
#3 rows had missing, incomplete, NaN data present 
Explanation: Question 3. Why would GRE have a larger STD than GPA?
Answer: The GRE variable has a larger 'std' value since the range of GRE scores varies from 220 to 800 while the range for GPA varies from 2.26 to 4.00. 
Question 4. Drop data points with missing data
End of explanation
#boxplot for GRE column data
df_raw.boxplot(column = 'gre', return_type = 'axes')
#boxplot for GPA column data
df_raw.boxplot(column = 'gpa', return_type = 'axes')
Explanation: Question 5. Confirm that you dropped the correct data. How can you tell?
Answer: Code in question one returned 400 observations across 4 rows. Culled data using '.dropna()' method returns 397 observational rows, implying that three rows had been removed due to NaN data being present.
Question 6. Create box plots for GRE and GPA
End of explanation
# distribution plot of 'admit' variable with mean
df_raw.admit.plot(kind = 'density', sharex = False, sharey = False, figsize = (10,4));plt.legend(loc='best')
# 
plt.vlines(df_raw.admit.mean(),     # Plot black line at mean
           ymin=0, 
           ymax=2.0,
           linewidth=4.0)
# distribution plot of 'gre' variable with mean
df_raw.gre.plot(kind = 'density', sharex = False, sharey = False, figsize = (10,4));plt.legend(loc='best')
# 
plt.vlines(df_raw.gre.mean(),     # Plot black line at mean
           ymin=0, 
           ymax=0.0035,
           linewidth=4.0)
# distribution plot of 'gpa' variable with mean
df_raw.gpa.plot(kind = 'density', sharex = False, sharey = False, figsize = (10,4));plt.legend(loc='best')
# 
plt.vlines(df_raw.gpa.mean(),     # Plot black line at mean
           ymin=0, 
           ymax=1.0,
           linewidth=4.0)
# distribution plot of 'prestige' variable with mean
df_raw.prestige.plot(kind = 'density', sharex = False, sharey = False, figsize = (10,4));plt.legend(loc='best')
# 
plt.vlines(df_raw.prestige.mean(),     # Plot black line at mean
           ymin=0, 
           ymax=0.6,
           linewidth=4.0)
Explanation: Question 7. What do this plots show?
Answer: 
GRE Boxplot:
    The mean for this variable lies just south of 600 (around 580) and the interquartile range lies between 650 and 510 as indicated by the blue square. The box plot displays a significant outlier at 300 which has not been included into the range as it falls well outside the acceptable standard deviation from the mean. Further, this value is below the lower extreme of variable GPA.
GPA Boxplot:
    The mean GPA value falls right at ~3.40 with the interquartile range falling between ~3.64 at the upper quartile and ~3.18 at the lower quartile. The lower extreme of this data is right at 2.4 while the upper extreme extends beyond 4.00 despite the maximum of this data being 4.00.
Question 8. Describe each distribution
End of explanation
# correlation matrix for variables in df_raw
df_raw.corr()
Explanation: Question 9.  If our model had an assumption of a normal distribution would we meet that requirement?
Answer: We would not meet that requirement as only the variable 'gre' displays itself in a quasi normal distribution. The variables for admit, gpa, and prestige are abnormally distributed.
Question 10.  Does this distribution need correction? If so, why? How?
Answer: Yes, this data does need to be corrected. If we are to compare these variables through linear regression or other statistics inferential tools, the data must be normalized in order to conform to a more normal distribution. 
We can accomplish this by using breating a new dataframe like so:
(df_norm = (df_raw - df_raw.mean()) / (df_raw.max() - df.min())
Sourced solution for normalization:
http://stackoverflow.com/questions/12525722/normalize-data-in-pandas
Question 11. Which of our variables are potentially colinear?
End of explanation
#utilized this stackoverflow.com resource to attempt to impute missing data 
#(http://stackoverflow.com/questions/21050426/pandas-impute-nans)
#data imputation for variable 'admit' 
#first commented out line of code will not run. Had errors with "keys" in ...df_raw.groupby('keys')...
#df_raw['admit'].fillna(df_raw.groupby('keys')['admit'].transform('mean'), inplace = True)
df_raw['admit'].fillna(df_raw['admit'].mean(), inplace = True)
Explanation: Question 12. What did you find?
Answer: 
    The strongest, most interesting correlation that exists between two variables exist between variables 'admit' and 'prestige'. The two variables are negatively correlated (-0.241). This would imply that as the prestige of your school increases by one unit your likelyhood of admission to UCLA decreases by a factor of -0.241 (or -25%), holding all other variables constant.
    GPA and GRE variables are positively correlated in that as your GPA increases, your GRE score increases by a factor of 0.382408. 
Question 13. Write an analysis plan for exploring the association between grad school admissions rates and prestige of  undergraduate schools.
Answer: 
    I will examine the relationship between variables 'prestige' and 'admit' using the admissions.csv database in order to determine if the two varibles are correlated and if they are causally linked. Further, I will determine if this relationship is statistically significant.
Question 14. What is your hypothesis?
Answer: 
H1 = There exists a statistically significant relationship between undergraduate school prestige ('prestige') and admission ('admit').
H0 = There exists an insignificant relationship between variables of undergraduate school prestige ('prestige') and admission ('admit').
Bonus/Advanced
1. Bonus: Explore alternatives to dropping obervations with missing data
2. Bonus: Log transform the skewed data
3. Advanced: Impute missing data
End of explanation
<END_TASK>