<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 <!--BOOK_INFORMATION-->
<img align="left" style="padding-right
Step1:  A Multiply Indexed Series
Let's start by considering how we might represent two-dimensional data within a one-dimensional Series.
For concreteness, we will consider a series of data where each point has a character and numerical key.
The bad way
Suppose you would like to track data about states from two different years.
Using the Pandas tools we've already covered, you might be tempted to simply use Python tuples as keys
Step2:  With this indexing scheme, you can straightforwardly index or slice the series based on this multiple index
Step3:  But the convenience ends there. For example, if you need to select all values from 2010, you'll need to do some messy (and potentially slow) munging to make it happen
Step4:  This produces the desired result, but is not as clean (or as efficient for large datasets) as the slicing syntax we've grown to love in Pandas.
The Better Way
Step5:  Notice that the MultiIndex contains multiple levels of indexing–in this case, the state names and the years, as well as multiple labels for each data point which encode these levels.
If we re-index our series with this MultiIndex, we see the hierarchical representation of the data
Step6:  Here the first two columns of the Series representation show the multiple index values, while the third column shows the data.
Notice that some entries are missing in the first column
Step7:  The result is a singly indexed array with just the keys we're interested in.
This syntax is much more convenient (and the operation is much more efficient!) than the home-spun tuple-based multi-indexing solution that we started with.
We'll now further discuss this sort of indexing operation on hieararchically indexed data.
MultiIndex as extra dimension
You might notice something else here
Step8:  Naturally, the stack() method provides the opposite operation
Step9:  Seeing this, you might wonder why would we would bother with hierarchical indexing at all.
The reason is simple
Step10:  In addition, all the ufuncs and other functionality discussed in Operating on Data in Pandas work with hierarchical indices as well.
Here we compute the fraction of people under 18 by year, given the above data
Step11:  This allows us to easily and quickly manipulate and explore even high-dimensional data.
Methods of MultiIndex Creation
The most straightforward way to construct a multiply indexed Series or DataFrame is to simply pass a list of two or more index arrays to the constructor. For example
Step12:  The work of creating the MultiIndex is done in the background.
Similarly, if you pass a dictionary with appropriate tuples as keys, Pandas will automatically recognize this and use a MultiIndex by default
Step13:  Nevertheless, it is sometimes useful to explicitly create a MultiIndex; we'll see a couple of these methods here.
Explicit MultiIndex constructors
For more flexibility in how the index is constructed, you can instead use the class method constructors available in the pd.MultiIndex.
For example, as we did before, you can construct the MultiIndex from a simple list of arrays giving the index values within each level
Step14:  You can construct it from a list of tuples giving the multiple index values of each point
Step15:  You can even construct it from a Cartesian product of single indices
Step16:  Similarly, you can construct the MultiIndex directly using its internal encoding by passing levels (a list of lists containing available index values for each level) and labels (a list of lists that reference these labels)
Step17:  Any of these objects can be passed as the index argument when creating a Series or Dataframe, or be passed to the reindex method of an existing Series or DataFrame.
MultiIndex level names
Sometimes it is convenient to name the levels of the MultiIndex.
This can be accomplished by passing the names argument to any of the above MultiIndex constructors, or by setting the names attribute of the index after the fact
Step18:  With more involved datasets, this can be a useful way to keep track of the meaning of various index values.
MultiIndex for columns
In a DataFrame, the rows and columns are completely symmetric, and just as the rows can have multiple levels of indices, the columns can have multiple levels as well.
Consider the following, which is a mock-up of some (somewhat realistic) medical data
Step19:  Here we see where the multi-indexing for both rows and columns can come in very handy.
This is fundamentally four-dimensional data, where the dimensions are the subject, the measurement type, the year, and the visit number.
With this in place we can, for example, index the top-level column by the person's name and get a full DataFrame containing just that person's information
Step20:  For complicated records containing multiple labeled measurements across multiple times for many subjects (people, countries, cities, etc.) use of hierarchical rows and columns can be extremely convenient!
Indexing and Slicing a MultiIndex
Indexing and slicing on a MultiIndex is designed to be intuitive, and it helps if you think about the indices as added dimensions.
We'll first look at indexing multiply indexed Series, and then multiply-indexed DataFrames.
Multiply indexed Series
Consider the multiply indexed Series of state populations we saw earlier
Step21:  We can access single elements by indexing with multiple terms
Step22:  The MultiIndex also supports partial indexing, or indexing just one of the levels in the index.
The result is another Series, with the lower-level indices maintained
Step23:  Partial slicing is available as well, as long as the MultiIndex is sorted (see discussion in Sorted and Unsorted Indices)
Step24:  With sorted indices, partial indexing can be performed on lower levels by passing an empty slice in the first index
Step25:  Other types of indexing and selection (discussed in Data Indexing and Selection) work as well; for example, selection based on Boolean masks
Step26:  Selection based on fancy indexing also works
Step27:  Multiply indexed DataFrames
A multiply indexed DataFrame behaves in a similar manner.
Consider our toy medical DataFrame from before
Step28:  Remember that columns are primary in a DataFrame, and the syntax used for multiply indexed Series applies to the columns.
For example, we can recover Guido's heart rate data with a simple operation
Step29:  Also, as with the single-index case, we can use the loc, iloc, and ix indexers introduced in Data Indexing and Selection. For example
Step30:  These indexers provide an array-like view of the underlying two-dimensional data, but each individual index in loc or iloc can be passed a tuple of multiple indices. For example
Step31:  Working with slices within these index tuples is not especially convenient; trying to create a slice within a tuple will lead to a syntax error
Step32:  You could get around this by building the desired slice explicitly using Python's built-in slice() function, but a better way in this context is to use an IndexSlice object, which Pandas provides for precisely this situation.
For example
Step33:  There are so many ways to interact with data in multiply indexed Series and DataFrames, and as with many tools in this book the best way to become familiar with them is to try them out!
Rearranging Multi-Indices
One of the keys to working with multiply indexed data is knowing how to effectively transform the data.
There are a number of operations that will preserve all the information in the dataset, but rearrange it for the purposes of various computations.
We saw a brief example of this in the stack() and unstack() methods, but there are many more ways to finely control the rearrangement of data between hierarchical indices and columns, and we'll explore them here.
Sorted and unsorted indices
Earlier, we briefly mentioned a caveat, but we should emphasize it more here.
Many of the MultiIndex slicing operations will fail if the index is not sorted.
Let's take a look at this here.
We'll start by creating some simple multiply indexed data where the indices are not lexographically sorted
Step34:  If we try to take a partial slice of this index, it will result in an error
Step35:  Although it is not entirely clear from the error message, this is the result of the MultiIndex not being sorted.
For various reasons, partial slices and other similar operations require the levels in the MultiIndex to be in sorted (i.e., lexographical) order.
Pandas provides a number of convenience routines to perform this type of sorting; examples are the sort_index() and sortlevel() methods of the DataFrame.
We'll use the simplest, sort_index(), here
Step36:  With the index sorted in this way, partial slicing will work as expected
Step37:  Stacking and unstacking indices
As we saw briefly before, it is possible to convert a dataset from a stacked multi-index to a simple two-dimensional representation, optionally specifying the level to use
Step38:  The opposite of unstack() is stack(), which here can be used to recover the original series
Step39:  Index setting and resetting
Another way to rearrange hierarchical data is to turn the index labels into columns; this can be accomplished with the reset_index method.
Calling this on the population dictionary will result in a DataFrame with a state and year column holding the information that was formerly in the index.
For clarity, we can optionally specify the name of the data for the column representation
Step40:  Often when working with data in the real world, the raw input data looks like this and it's useful to build a MultiIndex from the column values.
This can be done with the set_index method of the DataFrame, which returns a multiply indexed DataFrame
Step41:  In practice, I find this type of reindexing to be one of the more useful patterns when encountering real-world datasets.
Data Aggregations on Multi-Indices
We've previously seen that Pandas has built-in data aggregation methods, such as mean(), sum(), and max().
For hierarchically indexed data, these can be passed a level parameter that controls which subset of the data the aggregate is computed on.
For example, let's return to our health data
Step42:  Perhaps we'd like to average-out the measurements in the two visits each year. We can do this by naming the index level we'd like to explore, in this case the year
Step43:  By further making use of the axis keyword, we can take the mean among levels on the columns as well
<ASSISTANT_TASK:>
Python Code:
import pandas as pd
import numpy as np
Explanation: <!--BOOK_INFORMATION-->
<img align="left" style="padding-right:10px;" src="figures/PDSH-cover-small.png">
This notebook contains an excerpt from the Python Data Science Handbook by Jake VanderPlas; the content is available on GitHub.
The text is released under the CC-BY-NC-ND license, and code is released under the MIT license. If you find this content useful, please consider supporting the work by buying the book!
No changes were made to the contents of this notebook from the original.
<!--NAVIGATION-->
< Handling Missing Data | Contents | Combining Datasets: Concat and Append >
Hierarchical Indexing
Up to this point we've been focused primarily on one-dimensional and two-dimensional data, stored in Pandas Series and DataFrame objects, respectively.
Often it is useful to go beyond this and store higher-dimensional data–that is, data indexed by more than one or two keys.
While Pandas does provide Panel and Panel4D objects that natively handle three-dimensional and four-dimensional data (see Aside: Panel Data), a far more common pattern in practice is to make use of hierarchical indexing (also known as multi-indexing) to incorporate multiple index levels within a single index.
In this way, higher-dimensional data can be compactly represented within the familiar one-dimensional Series and two-dimensional DataFrame objects.
In this section, we'll explore the direct creation of MultiIndex objects, considerations when indexing, slicing, and computing statistics across multiply indexed data, and useful routines for converting between simple and hierarchically indexed representations of your data.
We begin with the standard imports:
End of explanation
index = [('California', 2000), ('California', 2010),
         ('New York', 2000), ('New York', 2010),
         ('Texas', 2000), ('Texas', 2010)]
populations = [33871648, 37253956,
               18976457, 19378102,
               20851820, 25145561]
pop = pd.Series(populations, index=index)
pop
Explanation: A Multiply Indexed Series
Let's start by considering how we might represent two-dimensional data within a one-dimensional Series.
For concreteness, we will consider a series of data where each point has a character and numerical key.
The bad way
Suppose you would like to track data about states from two different years.
Using the Pandas tools we've already covered, you might be tempted to simply use Python tuples as keys:
End of explanation
pop[('California', 2010):('Texas', 2000)]
Explanation: With this indexing scheme, you can straightforwardly index or slice the series based on this multiple index:
End of explanation
pop[[i for i in pop.index if i[1] == 2010]]
Explanation: But the convenience ends there. For example, if you need to select all values from 2010, you'll need to do some messy (and potentially slow) munging to make it happen:
End of explanation
index = pd.MultiIndex.from_tuples(index)
index
Explanation: This produces the desired result, but is not as clean (or as efficient for large datasets) as the slicing syntax we've grown to love in Pandas.
The Better Way: Pandas MultiIndex
Fortunately, Pandas provides a better way.
Our tuple-based indexing is essentially a rudimentary multi-index, and the Pandas MultiIndex type gives us the type of operations we wish to have.
We can create a multi-index from the tuples as follows:
End of explanation
pop = pop.reindex(index)
pop
Explanation: Notice that the MultiIndex contains multiple levels of indexing–in this case, the state names and the years, as well as multiple labels for each data point which encode these levels.
If we re-index our series with this MultiIndex, we see the hierarchical representation of the data:
End of explanation
pop[:, 2010]
Explanation: Here the first two columns of the Series representation show the multiple index values, while the third column shows the data.
Notice that some entries are missing in the first column: in this multi-index representation, any blank entry indicates the same value as the line above it.
Now to access all data for which the second index is 2010, we can simply use the Pandas slicing notation:
End of explanation
pop_df = pop.unstack()
pop_df
Explanation: The result is a singly indexed array with just the keys we're interested in.
This syntax is much more convenient (and the operation is much more efficient!) than the home-spun tuple-based multi-indexing solution that we started with.
We'll now further discuss this sort of indexing operation on hieararchically indexed data.
MultiIndex as extra dimension
You might notice something else here: we could easily have stored the same data using a simple DataFrame with index and column labels.
In fact, Pandas is built with this equivalence in mind. The unstack() method will quickly convert a multiply indexed Series into a conventionally indexed DataFrame:
End of explanation
pop_df.stack()
Explanation: Naturally, the stack() method provides the opposite operation:
End of explanation
pop_df = pd.DataFrame({'total': pop,
                       'under18': [9267089, 9284094,
                                   4687374, 4318033,
                                   5906301, 6879014]})
pop_df
Explanation: Seeing this, you might wonder why would we would bother with hierarchical indexing at all.
The reason is simple: just as we were able to use multi-indexing to represent two-dimensional data within a one-dimensional Series, we can also use it to represent data of three or more dimensions in a Series or DataFrame.
Each extra level in a multi-index represents an extra dimension of data; taking advantage of this property gives us much more flexibility in the types of data we can represent. Concretely, we might want to add another column of demographic data for each state at each year (say, population under 18) ; with a MultiIndex this is as easy as adding another column to the DataFrame:
End of explanation
f_u18 = pop_df['under18'] / pop_df['total']
f_u18.unstack()
Explanation: In addition, all the ufuncs and other functionality discussed in Operating on Data in Pandas work with hierarchical indices as well.
Here we compute the fraction of people under 18 by year, given the above data:
End of explanation
df = pd.DataFrame(np.random.rand(4, 2),
                  index=[['a', 'a', 'b', 'b'], [1, 2, 1, 2]],
                  columns=['data1', 'data2'])
df
Explanation: This allows us to easily and quickly manipulate and explore even high-dimensional data.
Methods of MultiIndex Creation
The most straightforward way to construct a multiply indexed Series or DataFrame is to simply pass a list of two or more index arrays to the constructor. For example:
End of explanation
data = {('California', 2000): 33871648,
        ('California', 2010): 37253956,
        ('Texas', 2000): 20851820,
        ('Texas', 2010): 25145561,
        ('New York', 2000): 18976457,
        ('New York', 2010): 19378102}
pd.Series(data)
Explanation: The work of creating the MultiIndex is done in the background.
Similarly, if you pass a dictionary with appropriate tuples as keys, Pandas will automatically recognize this and use a MultiIndex by default:
End of explanation
pd.MultiIndex.from_arrays([['a', 'a', 'b', 'b'], [1, 2, 1, 2]])
Explanation: Nevertheless, it is sometimes useful to explicitly create a MultiIndex; we'll see a couple of these methods here.
Explicit MultiIndex constructors
For more flexibility in how the index is constructed, you can instead use the class method constructors available in the pd.MultiIndex.
For example, as we did before, you can construct the MultiIndex from a simple list of arrays giving the index values within each level:
End of explanation
pd.MultiIndex.from_tuples([('a', 1), ('a', 2), ('b', 1), ('b', 2)])
Explanation: You can construct it from a list of tuples giving the multiple index values of each point:
End of explanation
pd.MultiIndex.from_product([['a', 'b'], [1, 2]])
Explanation: You can even construct it from a Cartesian product of single indices:
End of explanation
pd.MultiIndex(levels=[['a', 'b'], [1, 2]],
              labels=[[0, 0, 1, 1], [0, 1, 0, 1]])
Explanation: Similarly, you can construct the MultiIndex directly using its internal encoding by passing levels (a list of lists containing available index values for each level) and labels (a list of lists that reference these labels):
End of explanation
pop.index.names = ['state', 'year']
pop
Explanation: Any of these objects can be passed as the index argument when creating a Series or Dataframe, or be passed to the reindex method of an existing Series or DataFrame.
MultiIndex level names
Sometimes it is convenient to name the levels of the MultiIndex.
This can be accomplished by passing the names argument to any of the above MultiIndex constructors, or by setting the names attribute of the index after the fact:
End of explanation
# hierarchical indices and columns
index = pd.MultiIndex.from_product([[2013, 2014], [1, 2]],
                                   names=['year', 'visit'])
columns = pd.MultiIndex.from_product([['Bob', 'Guido', 'Sue'], ['HR', 'Temp']],
                                     names=['subject', 'type'])
# mock some data
data = np.round(np.random.randn(4, 6), 1)
data[:, ::2] *= 10
data += 37
# create the DataFrame
health_data = pd.DataFrame(data, index=index, columns=columns)
health_data
Explanation: With more involved datasets, this can be a useful way to keep track of the meaning of various index values.
MultiIndex for columns
In a DataFrame, the rows and columns are completely symmetric, and just as the rows can have multiple levels of indices, the columns can have multiple levels as well.
Consider the following, which is a mock-up of some (somewhat realistic) medical data:
End of explanation
health_data['Guido']
Explanation: Here we see where the multi-indexing for both rows and columns can come in very handy.
This is fundamentally four-dimensional data, where the dimensions are the subject, the measurement type, the year, and the visit number.
With this in place we can, for example, index the top-level column by the person's name and get a full DataFrame containing just that person's information:
End of explanation
pop
Explanation: For complicated records containing multiple labeled measurements across multiple times for many subjects (people, countries, cities, etc.) use of hierarchical rows and columns can be extremely convenient!
Indexing and Slicing a MultiIndex
Indexing and slicing on a MultiIndex is designed to be intuitive, and it helps if you think about the indices as added dimensions.
We'll first look at indexing multiply indexed Series, and then multiply-indexed DataFrames.
Multiply indexed Series
Consider the multiply indexed Series of state populations we saw earlier:
End of explanation
pop['California', 2000]
Explanation: We can access single elements by indexing with multiple terms:
End of explanation
pop['California']
Explanation: The MultiIndex also supports partial indexing, or indexing just one of the levels in the index.
The result is another Series, with the lower-level indices maintained:
End of explanation
pop.loc['California':'New York']
Explanation: Partial slicing is available as well, as long as the MultiIndex is sorted (see discussion in Sorted and Unsorted Indices):
End of explanation
pop[:, 2000]
Explanation: With sorted indices, partial indexing can be performed on lower levels by passing an empty slice in the first index:
End of explanation
pop[pop > 22000000]
Explanation: Other types of indexing and selection (discussed in Data Indexing and Selection) work as well; for example, selection based on Boolean masks:
End of explanation
pop[['California', 'Texas']]
Explanation: Selection based on fancy indexing also works:
End of explanation
health_data
Explanation: Multiply indexed DataFrames
A multiply indexed DataFrame behaves in a similar manner.
Consider our toy medical DataFrame from before:
End of explanation
health_data['Guido', 'HR']
Explanation: Remember that columns are primary in a DataFrame, and the syntax used for multiply indexed Series applies to the columns.
For example, we can recover Guido's heart rate data with a simple operation:
End of explanation
health_data.iloc[:2, :2]
Explanation: Also, as with the single-index case, we can use the loc, iloc, and ix indexers introduced in Data Indexing and Selection. For example:
End of explanation
health_data.loc[:, ('Bob', 'HR')]
Explanation: These indexers provide an array-like view of the underlying two-dimensional data, but each individual index in loc or iloc can be passed a tuple of multiple indices. For example:
End of explanation
health_data.loc[(:, 1), (:, 'HR')]
Explanation: Working with slices within these index tuples is not especially convenient; trying to create a slice within a tuple will lead to a syntax error:
End of explanation
idx = pd.IndexSlice
health_data.loc[idx[:, 1], idx[:, 'HR']]
Explanation: You could get around this by building the desired slice explicitly using Python's built-in slice() function, but a better way in this context is to use an IndexSlice object, which Pandas provides for precisely this situation.
For example:
End of explanation
index = pd.MultiIndex.from_product([['a', 'c', 'b'], [1, 2]])
data = pd.Series(np.random.rand(6), index=index)
data.index.names = ['char', 'int']
data
Explanation: There are so many ways to interact with data in multiply indexed Series and DataFrames, and as with many tools in this book the best way to become familiar with them is to try them out!
Rearranging Multi-Indices
One of the keys to working with multiply indexed data is knowing how to effectively transform the data.
There are a number of operations that will preserve all the information in the dataset, but rearrange it for the purposes of various computations.
We saw a brief example of this in the stack() and unstack() methods, but there are many more ways to finely control the rearrangement of data between hierarchical indices and columns, and we'll explore them here.
Sorted and unsorted indices
Earlier, we briefly mentioned a caveat, but we should emphasize it more here.
Many of the MultiIndex slicing operations will fail if the index is not sorted.
Let's take a look at this here.
We'll start by creating some simple multiply indexed data where the indices are not lexographically sorted:
End of explanation
try:
    data['a':'b']
except KeyError as e:
    print(type(e))
    print(e)
Explanation: If we try to take a partial slice of this index, it will result in an error:
End of explanation
data = data.sort_index()
data
Explanation: Although it is not entirely clear from the error message, this is the result of the MultiIndex not being sorted.
For various reasons, partial slices and other similar operations require the levels in the MultiIndex to be in sorted (i.e., lexographical) order.
Pandas provides a number of convenience routines to perform this type of sorting; examples are the sort_index() and sortlevel() methods of the DataFrame.
We'll use the simplest, sort_index(), here:
End of explanation
data['a':'b']
Explanation: With the index sorted in this way, partial slicing will work as expected:
End of explanation
pop.unstack(level=0)
pop.unstack(level=1)
Explanation: Stacking and unstacking indices
As we saw briefly before, it is possible to convert a dataset from a stacked multi-index to a simple two-dimensional representation, optionally specifying the level to use:
End of explanation
pop.unstack().stack()
Explanation: The opposite of unstack() is stack(), which here can be used to recover the original series:
End of explanation
pop_flat = pop.reset_index(name='population')
pop_flat
Explanation: Index setting and resetting
Another way to rearrange hierarchical data is to turn the index labels into columns; this can be accomplished with the reset_index method.
Calling this on the population dictionary will result in a DataFrame with a state and year column holding the information that was formerly in the index.
For clarity, we can optionally specify the name of the data for the column representation:
End of explanation
pop_flat.set_index(['state', 'year'])
Explanation: Often when working with data in the real world, the raw input data looks like this and it's useful to build a MultiIndex from the column values.
This can be done with the set_index method of the DataFrame, which returns a multiply indexed DataFrame:
End of explanation
health_data
Explanation: In practice, I find this type of reindexing to be one of the more useful patterns when encountering real-world datasets.
Data Aggregations on Multi-Indices
We've previously seen that Pandas has built-in data aggregation methods, such as mean(), sum(), and max().
For hierarchically indexed data, these can be passed a level parameter that controls which subset of the data the aggregate is computed on.
For example, let's return to our health data:
End of explanation
data_mean = health_data.mean(level='year')
data_mean
Explanation: Perhaps we'd like to average-out the measurements in the two visits each year. We can do this by naming the index level we'd like to explore, in this case the year:
End of explanation
data_mean.mean(axis=1, level='type')
Explanation: By further making use of the axis keyword, we can take the mean among levels on the columns as well:
End of explanation
<END_TASK>