<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 Example
Step1:  Now we should have a data/names directory which contains a number of text files, one for each year of data
Step2:  Let's take a quick look at one of these files
Step3:  Each file is just a comma-separated list of names, genders, and counts of babies with that name in each year.
We can load these files using pd.read_csv, which is specifically designed for this
Step4:  Oops! Something went wrong. Our algorithm tried to be smart, and use the first line as index labels.
Let's fix this by specifying the index names manually
Step5:  That looks better. Now we can start playing with the data a bit.
GroupBy
Step6:  Now we can take the sum of the births for each of these
Step7:  But there's an easier way to do this, using one of Pandas' very powerful features
Step8:  This grouped object is now an abstract representation of the data, where the data is split on the given column.
In order to actually do something with this data, we need to specify an aggregation operation to do across the data.
In this case, what we want is the sum
Step9:  We can do other aggregations as well
Step10:  Or, if we wish, we can get a description of the grouping
Step11:  Concatenating multiple data sources
But here we've just been looking at a single year. Let's try to put together all the data in all the years.
To do this, we'll have to use pandas concat function to concatenate all the data together.
First we'll create a function which loads the data as we did the above data
Step12:  Now let's load all the data into a list, and call pd.concat on that list
Step13:  It looks like we've done it!
Let's start with something easy
Step14:  We can use the plot() method to see a quick plot of these (note that because we used the %matplotlib inline magic at the start of the notebook, the resulting plot will be shown inline within the notebook).
Step15:  The so-called "baby boom" generation after the second world war is abundantly clear!
We can also use other aggregates
Step16:  Apparently there's been a huge increase of the diversity of names with time!
groupby can also be used to add columns to the data
Step17:  Notice that the apply() function iterates over each group, and calls a function which modifies the group.
This result is then re-constructed into a container which looks ike the original dataframe.
Pivot Tables
Next we'll discuss Pivot Tables, which are an even more powerful way of (re)organizing your data.
Let's say that we want to plot the men and women separately.
We could do this by using masking, as follows
Step18:  And then we could proceed as above, using groupby to group on the year.
But we would end up with two different views of the data. A better way to do this is to use a pivot_table, which is essentially a groupby in multiple dimensions at once
Step19:  Note that this has grouped the index by the value of year, and grouped the columns by the value of gender.
Let's plot the results now
Step20:  Name Evolution Over Time
Some names have shifted from being girls names to being boys names. Let's take a look at some of these
Step21:  We can see that prior to about 1905, all babies named Allison were male. Over the 20th century, this reversed, until the end of the century nearly all Allisons were female!
There's some noise in this data
<ASSISTANT_TASK:>
Python Code:
# !curl -O http://www.ssa.gov/oact/babynames/names.zip
# !mkdir -p data/names
# !mv names.zip data/names/
# !cd data/names/ && unzip names.zip
Explanation: Example: Names in the Wild
This example is drawn from Wes McKinney's excellent book on the Pandas library, O'Reilly's Python for Data Analysis.
We'll be taking a look at a freely available dataset: the database of names given to babies in the United States over the last century.
First things first, we need to download the data, which can be found at http://www.ssa.gov/oact/babynames/limits.html.
If you uncomment the following commands, it will do this automatically (note that these are linux shell commands; they will not work on Windows):
End of explanation
!ls data/names
Explanation: Now we should have a data/names directory which contains a number of text files, one for each year of data:
End of explanation
!head data/names/yob1880.txt
Explanation: Let's take a quick look at one of these files:
End of explanation
names1880 = pd.read_csv('data/names/yob1880.txt')
names1880.head()
Explanation: Each file is just a comma-separated list of names, genders, and counts of babies with that name in each year.
We can load these files using pd.read_csv, which is specifically designed for this:
End of explanation
names1880 = pd.read_csv('data/names/yob1880.txt',
                        names=['name', 'gender', 'births'])
names1880.head()
Explanation: Oops! Something went wrong. Our algorithm tried to be smart, and use the first line as index labels.
Let's fix this by specifying the index names manually:
End of explanation
males = names1880[names1880.gender == 'M']
females = names1880[names1880.gender == 'F']
Explanation: That looks better. Now we can start playing with the data a bit.
GroupBy: aggregates on values
First let's think about how we might count the total number of females and males born in the US in 1880.
If you're used to NumPy, you might be tempted to use masking like this:
First, we can get a mask over all females & males, and then use it to select a subset of the data:
End of explanation
males.births.sum(), females.births.sum()
Explanation: Now we can take the sum of the births for each of these:
End of explanation
grouped = names1880.groupby('gender')
grouped
Explanation: But there's an easier way to do this, using one of Pandas' very powerful features: groupby:
End of explanation
grouped.sum()
Explanation: This grouped object is now an abstract representation of the data, where the data is split on the given column.
In order to actually do something with this data, we need to specify an aggregation operation to do across the data.
In this case, what we want is the sum:
End of explanation
grouped.size()
grouped.mean()
Explanation: We can do other aggregations as well:
End of explanation
grouped.describe()
Explanation: Or, if we wish, we can get a description of the grouping:
End of explanation
def load_year(year):
    data = pd.read_csv('data/names/yob{0}.txt'.format(year),
                       names=['name', 'gender', 'births'])
    data['year'] = year
    return data
Explanation: Concatenating multiple data sources
But here we've just been looking at a single year. Let's try to put together all the data in all the years.
To do this, we'll have to use pandas concat function to concatenate all the data together.
First we'll create a function which loads the data as we did the above data:
End of explanation
names = pd.concat([load_year(year) for year in range(1880, 2014)])
names.head()
Explanation: Now let's load all the data into a list, and call pd.concat on that list:
End of explanation
births = names.groupby('year').births.sum()
births.head()
Explanation: It looks like we've done it!
Let's start with something easy: we'll use groupby again to see the total number of births per year:
End of explanation
births.plot();
Explanation: We can use the plot() method to see a quick plot of these (note that because we used the %matplotlib inline magic at the start of the notebook, the resulting plot will be shown inline within the notebook).
End of explanation
names.groupby('year').births.count().plot();
Explanation: The so-called "baby boom" generation after the second world war is abundantly clear!
We can also use other aggregates: let's see how many names are used each year:
End of explanation
def add_frequency(group):
    group['birth_freq'] = group.births / group.births.sum()
    return group
names = names.groupby(['year', 'gender']).apply(add_frequency)
names.head()
Explanation: Apparently there's been a huge increase of the diversity of names with time!
groupby can also be used to add columns to the data: think of it as a view of the data that you're modifying. Let's add a column giving the frequency of each name within each year & gender:
End of explanation
men = names[names.gender == 'M']
women = names[names.gender == 'W']
Explanation: Notice that the apply() function iterates over each group, and calls a function which modifies the group.
This result is then re-constructed into a container which looks ike the original dataframe.
Pivot Tables
Next we'll discuss Pivot Tables, which are an even more powerful way of (re)organizing your data.
Let's say that we want to plot the men and women separately.
We could do this by using masking, as follows:
End of explanation
births = names.pivot_table('births',
                           index='year', columns='gender',
                           aggfunc=sum)
births.head()
Explanation: And then we could proceed as above, using groupby to group on the year.
But we would end up with two different views of the data. A better way to do this is to use a pivot_table, which is essentially a groupby in multiple dimensions at once:
End of explanation
births.plot(title='Total Births');
Explanation: Note that this has grouped the index by the value of year, and grouped the columns by the value of gender.
Let's plot the results now:
End of explanation
names_to_check = ['Allison', 'Alison']
# filter on just the names we're interested in
births = names[names.name.isin(names_to_check)]
# pivot table to get year vs. gender
births = births.pivot_table('births', index='year', columns='gender')
# fill all NaNs with zeros
births = births.fillna(0)
# normalize along columns
births = births.div(births.sum(1), axis=0)
births.plot(title='Fraction of babies named Allison');
Explanation: Name Evolution Over Time
Some names have shifted from being girls names to being boys names. Let's take a look at some of these:
End of explanation
pd.rolling_mean(births, 5).plot(title="Allisons: 5-year moving average");
Explanation: We can see that prior to about 1905, all babies named Allison were male. Over the 20th century, this reversed, until the end of the century nearly all Allisons were female!
There's some noise in this data: we can smooth it out a bit by using a 5-year rolling mean:
End of explanation
<END_TASK>