<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 Set up
Use conda to install seaborn by running in a terminal
conda install seaborn
Step1:  What problem does pandas solve?
Python has long been great for data munging and preparation, but less so for data analysis and modeling. pandas helps fill this gap, enabling you to carry out your entire data analysis workflow in Python without having to switch to a more domain specific language like R.
Combined with the excellent IPython toolkit and other libraries, the environment for doing data analysis in Python excels in performance, productivity, and the ability to collaborate.
pandas does not implement significant modeling functionality outside of linear and panel regression; for this, look to statsmodels and scikit-learn. More work is still needed to make Python a first class statistical modeling environment, but we are well on our way toward that goal.
http
Step2:  Load a data set
"Census Income" dataset
http
Step3:  pandas can load a lot more than csvs, this tutorial shows how pandas can read excel, sql,
and even copy and paste...
http
Step4:  Strip spaces in columns
Step5:  for more info on lambda and map
http
Step6:  Find nulls
Step7:  Fill nulls
Step8:  Selecting rows and columns
Step9:  Differences between .loc, .iloc, and .ix
http
Step10:  Rename columns
Step11:  Boolean
Step12:  Groupby
Step13:  plotting with pandas
Step14:  Practice Exercises
Step15:  Extra practice
Step16:  http
<ASSISTANT_TASK:>
Python Code:
# imports a library 'pandas', names it as 'pd'
import pandas as pd
import numpy as np
import seaborn as sns
import matplotlib.pyplot as plt
# enables inline plots, without it plots don't show up in the notebook
%matplotlib inline
Explanation: Set up
Use conda to install seaborn by running in a terminal
conda install seaborn
End of explanation
# various options in pandas
pd.set_option('display.max_columns', None)
pd.set_option('display.max_rows', 20)
pd.set_option('display.precision', 3)
Explanation: What problem does pandas solve?
Python has long been great for data munging and preparation, but less so for data analysis and modeling. pandas helps fill this gap, enabling you to carry out your entire data analysis workflow in Python without having to switch to a more domain specific language like R.
Combined with the excellent IPython toolkit and other libraries, the environment for doing data analysis in Python excels in performance, productivity, and the ability to collaborate.
pandas does not implement significant modeling functionality outside of linear and panel regression; for this, look to statsmodels and scikit-learn. More work is still needed to make Python a first class statistical modeling environment, but we are well on our way toward that goal.
http://pandas.pydata.org/
End of explanation
# download the data and name the columns
cols = ['age', 'workclass', 'fnlwgt', 'education', 'education_num',
        'marital_status', 'occupation', 'relationship', 'ethnicity',
        'gender', 'capital_gain', 'capital_loss', 'hours_per_week',
        'country_of_origin', 'income']
df = pd.read_csv('http://archive.ics.uci.edu/ml/machine-learning-databases/adult/adult.data',
                       names = cols)
Explanation: Load a data set
"Census Income" dataset
http://archive.ics.uci.edu/ml/
End of explanation
# we can see there are no null values
# columns with numberical values are type int64, no need to set data type
df.info()
# to view the first 5 or specify with ex: .head(10)
df.head()
# there's a space before each string in this data
df.education.unique()
# looks like it's in every object column
df.workclass.unique()
Explanation: pandas can load a lot more than csvs, this tutorial shows how pandas can read excel, sql,
and even copy and paste...
http://www.gregreda.com/2013/10/26/intro-to-pandas-data-structures/
View
End of explanation
# loop through each column and strip all the spaces
for col in df:
    if df[col].dtype == 'O':
        df[col] = df[col].map(lambda x: x.strip(' '))
# Here's a break down of what that for loop is doing
# loops through df and gets the column names
for col in df:
    print col
# gets the column type
df.education.dtype
# if True then applys the map function
df.education.dtype == object
# strip function
x = ' string'
x.strip(' ')
# lambda creates a 'throw away' or 'anonymous' function
strip_string = lambda x: x.strip(' ')
strip_string(' string')
# same as this
def strip_string2(x):
    x = x.strip(' ')
    return x
strip_string2(' string')
# map applies the function to each item in the data frame column so
df[col].map(lambda x: x.strip(' '))
# does the same thing as
df['workclass'].map(strip_string2)
# but in the first case we don't have to define and name a function
Explanation: Strip spaces in columns
End of explanation
df.education.value_counts()
df.hours_per_week.mean()
df[['age', 'capital_gain', 'capital_loss', 'hours_per_week']].describe()
Explanation: for more info on lambda and map
http://www.python-course.eu/lambda.php
Descriptive
End of explanation
# as we saw with df.info() there are no nulls... 
# but if there were this would find the rows where age is null
df[df.age.isnull()]
# you could drop all those rows with
df_no_nulls = df[df.age.notnull()]
Explanation: Find nulls
End of explanation
null_df = pd.DataFrame([1,2,4,np.nan], columns = ['column1'])
null_df
# you can also fill nulls with a value or string
null_df.column1.fillna(1000)
null_df.column1.fillna(null_df.column1.median())
null_df.column1.fillna('string')
Explanation: Fill nulls
End of explanation
# select a row
df.iloc[0]
# select a range of rows
df.iloc[10:15]
# last 2 rows
df.iloc[-2:]
# selecting every other row in columns 3-5
df.iloc[::2, 2:5].head()
df.loc[0:2, ['age', 'relationship ']]
Explanation: Selecting rows and columns
End of explanation
# pd.DataFrame let's you turn series, arrays, lists, and more into data frame structures
df_index = pd.DataFrame([[1,2,3,4],[2,4,6,8],[3,5,7,9]], [11,13,12], columns = ['A', 'C', 'D', 'B'])
df_index
# iloc indexes by postion, not by the labels in the index
df_index.iloc[0:1]
# with loc both the start and the stop are included
df_index.loc[11:12]
# select columns by position
df_index.iloc[:,0:1]
# or by label
df_index.loc[:,'A':'D']
# ix: primarily label based, but will fall back to integer positional access
df_index.ix[:,'A':'C']
# ix: primarily label based, but will fall back to integer positional access
df_index.ix[:,0:2]
Explanation: Differences between .loc, .iloc, and .ix
http://pandas.pydata.org/pandas-docs/stable/indexing.html
by label
.loc[]
by integer position
.iloc[]
for both
.ix[]
End of explanation
df.columns
# replace a column name
df.rename(columns = {'native_country' : 'country_of_origin'}, inplace = False)
Explanation: Rename columns
End of explanation
df.native_country.unique()
df[df.native_country == 'United-States'].head()
df[(df.native_country != 'United-States') & (df.education_num > 9)].head()
df[(df.native_country != 'United-States') & (df.education_num > 9)].income.value_counts()
Explanation: Boolean
End of explanation
# How to groupby column and apply a function like sum, count, or mean
df.groupby(['relationship']).mean()
# To groupby multiple columns with multiple functions attached
df.groupby(['income', 'native_country']).age.agg(['count', 'mean'])
# grouped in order of which column is listed first
# combine groupby with boolean
df[df.native_country != 'United-States'].groupby(['education']).hours_per_week.mean()
Explanation: Groupby
End of explanation
df.age.hist(bins = 18);
# split the histograms by another column (with relatively few unique values)
df.hours_per_week.hist(by = df.income, bins = 25, figsize = (10,5));
# use value_counts() and a bar plot
df['workclass'].value_counts().plot(kind = 'bar')
df.boxplot(['age'], by = 'relationship');
df.plot(kind='scatter',
        x = 'age',
        y ='hours_per_week',
        alpha = .25,
        figsize = (10,5))
Explanation: plotting with pandas
End of explanation
# how many males and females are in this data set
# plot the total number of people in each occupation
# what is the lowest average age of an occupation
# create a boxplot of hours per week by education level
# create a new column for income where >50K = 1 and <=50K = 0
# hint... http://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.replace.html
# find which "native_country" has the highest percent of people earning >50K
# visualize what percent of people earn over 50K by education level
Explanation: Practice Exercises
End of explanation
# make a hexbin plot
Explanation: Extra practice
End of explanation
# check out plotting with Seaborn
Explanation: http://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.plot.html
End of explanation
<END_TASK>