<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 Introduction to data munging with Jupyter and pandas
PyGotham 2015
Step1:  The case for open source data tools
Reproducibility and Transparency
Cost -- compare capabilities between software you already use and open source here
Allows a diversity of platforms on a given team
The case for notebooks
They're amenable to sketching, and they're amenable to reproducibility.
You can retrace your own steps and also make a narrative for someone else to follow.
Built-in documentation improves workflow; magic methods anticipate data analysis needs.
IPython/Jupyter Notebook
Step2:  2. built-in documentation
shift-tab brings up brief function documentation
Step3:  3. markup!
Markdown and HTML
<img src='http
Step4:  5. multimedia
Step5:  6. Sharing! Notebooks are now viewable directly on github, and also exportable as PDF or HTML.
7. Shell commands!
Step6:  8. Keyboard shortcuts! Your implementation may vary.
Use what's here or roll your own.
pandas
Input
Source
Step7:  Transformation
Step8:  A WILD ZIP CODE APPEARS!
<img src="http
Step9:  Keep in mind you have options, and use magic methods to test implementation inline!
Step10:  Merging! Better than in traffic!
Group by knows aggregating strings is nonsensical, and so drops those.
But let's add state information to the mix again!
<ASSISTANT_TASK:>
Python Code:
from __future__ import division
import matplotlib as mpl
import matplotlib.pyplot as plt
import numpy as np
import pandas as pd
import seaborn as sns
import rpy2
from IPython.display import display, Image, YouTubeVideo
%matplotlib inline
Explanation: Introduction to data munging with Jupyter and pandas
PyGotham 2015
End of explanation
pd.re
from math import 
Explanation: The case for open source data tools
Reproducibility and Transparency
Cost -- compare capabilities between software you already use and open source here
Allows a diversity of platforms on a given team
The case for notebooks
They're amenable to sketching, and they're amenable to reproducibility.
You can retrace your own steps and also make a narrative for someone else to follow.
Built-in documentation improves workflow; magic methods anticipate data analysis needs.
IPython/Jupyter Notebook: some cool tips for beginners
1. holy moly tab completion
End of explanation
?pd.read_csv
Explanation: 2. built-in documentation
shift-tab brings up brief function documentation
End of explanation
%quickref
%lsmagic
%load_ext rmagic
%R x <- c(0:10, 50)
%R xm <- mean(x)
%R c(xm, mean(x, trim = 0.10))
Explanation: 3. markup!
Markdown and HTML
<img src='http://i.imgur.com/WypQf94.gif' align='left'></span>
<br>
<br>
<br>
LaTeX
<span style="font-size: 24px"> $\bar{x} = \frac{\sum_{i=1}^{n}w_i\cdot x_i}{\sum_{i=1}^{n}w_i}$</span>
Syntax highlighting for other languages
R
x &lt;- c(0:10, 50)
xm &lt;- mean(x)
c(xm, mean(x, trim = 0.10))
4. magic methods
End of explanation
YouTubeVideo("L4Hbv4ugUWk")
Explanation: 5. multimedia
End of explanation
!ls
Explanation: 6. Sharing! Notebooks are now viewable directly on github, and also exportable as PDF or HTML.
7. Shell commands!
End of explanation
?pd.read_csv()
# read in a CSV
# specify that zipcode should be treated as a string rather than an int!
AGI = pd.read_csv('12zpallagi.csv',dtype={'zipcode': str})
AGI.info()
Explanation: 8. Keyboard shortcuts! Your implementation may vary.
Use what's here or roll your own.
pandas
Input
Source: IRS.gov
End of explanation
# you can select columns by label or position!
AGI_column_subset = AGI[['STATE','AGI_STUB','zipcode','N1','A00100']]
# get information about type for a given field, and how many values you can expect for each
AGI_column_subset.info()
AGI_column_subset.describe()
# note this is inclusive!
AGI_row_subset = AGI_column_subset.ix[6:11]
AGI_row_subset
AGI_column_subset.rename(columns={'N1':'population','A00100':'amount'},inplace=True)
AGI_column_subset.head()
# group by zipcode and sum other values, resetting index
AGI_grouped = AGI_column_subset.groupby('zipcode').sum().reset_index()
AGI_grouped.head()
Explanation: Transformation
End of explanation
AGI_grouped['population'].mean()
#this can also be done using the na_values param upon being read in
null_zips = (AGI_grouped['zipcode'] == '00000')
AGI_grouped.loc[null_zips, 'zipcode'] = np.nan
AGI_grouped.head()
AGI_notnull = AGI_grouped.dropna()
AGI_notnull['population'].mean()
AGI_grouped.dropna(inplace=True)
# make a new column with the real amount, not in thousands
AGI_grouped['actual_amount'] = AGI_grouped['amount'] * 1000
Explanation: A WILD ZIP CODE APPEARS!
<img src="http://ecdn.funzypics.com/grumpycatmemes/pics/16/OH-No--The-Cute-Kitty-Cats-Are-Attacking-Like-Aliens----The-Hunt-Is-On-Wow-Quest-Guide-.jpg" align="left">
Delete or render null? You decide!
End of explanation
%timeit applied = AGI_grouped['amount'].apply(lambda x: x * 1000)
#being vectorized operations, this is happening at the C level and thereby much faster
%timeit vectorized = AGI_grouped['amount'] * 1000
AGI_grouped
# make a mean, using standard math operations!
AGI_grouped['weighted_mean_AGI'] = AGI_grouped['actual_amount']/AGI_grouped['population']
#use anonymous functions to change every value in a column!
#because this is an apply, much slower
AGI_grouped['weighted_mean_AGI']= AGI_grouped['weighted_mean_AGI'].apply(lambda x: round(x, 0))
AGI_grouped.info()
AGI_grouped.describe()
# drop columns you won't need
AGI_grouped.drop(['AGI_STUB','amount','actual_amount'],axis=1,inplace=True)
AGI_grouped.head()
Explanation: Keep in mind you have options, and use magic methods to test implementation inline!
End of explanation
# also look into pandas.Series.unique
AGI_subset_geo = AGI[['zipcode','STATE']].drop_duplicates()
AGI_subset_geo.head()
#merge rather than join if you want to use a common column other than the index
AGI_final = pd.merge(AGI_grouped, AGI_subset_geo, how='left', on='zipcode')
AGI_final.head()
# this gives you the greatest weighted_mean_AGI first
AGI_final.sort('weighted_mean_AGI',ascending=False).head()
# chain methods!
AGI_final.groupby('STATE').mean().sort('weighted_mean_AGI',ascending=False)
AGI_final.sort('weighted_mean_AGI').head().plot(kind='bar')
Explanation: Merging! Better than in traffic!
Group by knows aggregating strings is nonsensical, and so drops those.
But let's add state information to the mix again!
End of explanation
<END_TASK>