<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 Pandas
Pandas is a high-performance python library that provides a comprehensive set of data structures for manipulating tabular data, providing high-performance indexing, automatic alignment, reshaping, grouping, joining and statistical analyses capabilities.
The two primary data structures in pandas are the Series and the DataFrame objects.
Series Object
The Series object is the fundamental building block of pandas. A Series represents an one-dimensional array based on the NumPy ndarray but with a labeled index that significantly helps to access the elements.
A Series always has an index even if one is not specified, by default pandas will create an index that consists of sequential integers starting from zero. Access to elements is not by integer position but using values in the index referred as Labels.
Importing pandas into the application is simple. It is common to import both pandas and numpy with their objects mapped into the pd and np namespaces respectively.
Step1:  Creating Series
A Series can be created and initialised by passing either a scalar value, a NumPy nd array, a Python list or a Python Dict as the data parameter of the Series constructor.
Step2:  '0' is the index and '1' is the value. The data type (dtype) is also shown. We can also retrieve the value using the associated index.
Step3:  Creating Series with named index
Pandas will create different index types based on the type of data identified in the index parameter. These different index types are optimized to perform indexing operations for that specific data type. To specify the index at the time of creation of the Series, use the index parameter of the constructor.
Step4:  Please note the type of the index items. It is not string but 'object'.
Step5:  It is a common practice to initialize the Series objects using NumPy ndarrays, and with various NumPy functions that create arrays. The following code creates a Series from five normally distributed values
Step6:  A Series can also be created from a Python dictionary. The keys of the dictionary are used as the index lables for the Series
Step7:  Size, Shape, Count and Uniqueness of Values
Step8:  Peeking at data with heads, tails and take
pandas provides the .head() and .tail() methods to examine just the first few or last records in a Series. By default, these return the first five or last rows respectively, but you can use the n parameter or just pass an integer to specify the number of rows
Step9:  The .take() method will return the rows in a series that correspond to the zero-based positions specified in a list
Step10:  Looking up values in Series
Values in a Series object can be retrieved using the [] operator and passing either a single index label or a list of index labels.
Step11:  To alleviate the potential confusion in determining the label-based lookups versus position-based lookups, index based lookup can be enforced using the .loc[] accessor
Step12:  Lookup by position can be enforced using the iloc[] accessor
Step13:  If a location / position passed to .iloc[] in a list is out of bounds, an exception will be thrown. This is different than with .loc[], which if passed a label that does not exist, will return NaN as the value for that label
Step14:  A Series also has a property .ix that can be used to look up items either by label or by zero-based array position.
Step15:  This can become complicated if the indexes are integers and you pass a list of integers to ix. Since they are of the same type, the lookup will be by index label instead of position
Step16:  Alignment via index labels
A fundamental difference between a NumPy ndarray and a pandas Series is the ability of a Series to automatically align data from another Series based on label values before performing an operation.
Step17:  This is a very different result that what it would have been if it were two pure NumPy arrays being added. A NumPy ndarray would add the items in identical positions of each array resulting in different values.
Step18:  The process of adding two Series objects differs from the process of addition of arrays as it first aligns data based on index label values instead of simply applying the operation to elements in the same position. This becomes significantly powerful when using pandas Series to combine data based on labels instead of having to first order the data manually.
Arithmetic Operations
Arithemetic Operations <pre>(+,-,*,/)</pre> can be applied either to a Series or between 2 Series objects
Step19:  To reinforce the point that alignment is being performed when applying arithmetic operations across two Series objects, look at the following two Series as examples
Step20:  The reason for the above result is that during alignment, pandas actually performs a cartesian product of the sets of all the unique index labels in both Series objects, and then applies the specified operation on all items in the products.
To explain why there are four 'a' index values s10 contains two 'a' labels and s11 also contains two 'a' labels. Every combination of 'a' labels in each will be calculated resulting in four 'a' labels. There is one 'b' label from s10 and one 'c' label from s11. Since there is no matching label for either in the other Series object, they only result in a sing row in the resulting Series object.
Each combination of values for 'a' in both Series are computed, resulting in the four values
Step21:  Boolean selection
Items in a Series can be selected, based on the value instead of index labels, via the utilization of a Boolean selection.
Step22:  The result of these logical expressions is a Boolean selection, a Series of True and False values. The .sum() method of a Series, when given a series of Boolean values, will treat True as 1 and False as 0. The following demonstrates using this to determine the number of items in a Series that satisfy a given expression
Step23:  Reindexing a Series
Reindexing in pandas is a process that makes the data in a Series or DataFrame match a given set of labels.
This process of performing a reindex includes the following steps
Step24:  Greater flexibility in creating a new index is provided using the .reindex() method. An example of the flexibility of .reindex() over assigning the .index property directly is that the list provided to .reindex() can be of a different length than the number of rows in the Series
Step25:  There are several things here that are important to point out about .reindex() method.
First is that the result of .reindex() method is a new Series. This new Series has an index with labels that are provided as parameter to reindex().
For each item in the given parameter list, if the original Series contains that label, then the value is assigned to that label.
If that label does not exist in the original Series, pandas assigns a NaN value.
Rows in the Series without a label specified in the parameter of .reindex() is not included in the result.
To demonstrate that the result of .reindex() is a new Series object, changing a value in s2 does not change the values in s1
Step26:  Reindex is also useful when you want to align two Series to perform an operation on matching elements from each series; however, for some reason, the two Series has index labels that will not initially align.
Step27:  The reason why this happens in pandas are as follows
Step28:  The default action of inserting NaN as a missing value during reindexing can be changed by using the fill_value parameter of the method.
Step29:  When performing a reindex on ordered data such as a time series, it is possible to perform interpolation or filling of values. The following example demonstrates forward filling, often referred to as "last known value".
Step30:  Modifying a Series in-place
There are several ways that an existing Series can be modified in-place having either its values changed or having rows added or deleted.
A new item can be added to a Series by assigning a value to an index label that does not already exist.
Step31:  Items can be removed from a Series using the del() function and passing the index label(s) to be removed.
Step32:  Slicing a Series
Step33:  An important thing to keep in mind when using slicing, is that the result of the slice is actually a view into the original Series. Modification of values through the result of the slice will modify the original Series.
Step34:  Now the assignment of a value to an element of a slice will change the value in the original Series
Step35:  Slicing can be performed on Series objects with a non-integer index.
<ASSISTANT_TASK:>
Python Code:
import numpy as np
import pandas as pd
pd.__version__
np.__version__
# set some options to control output display
pd.set_option('display.notebook_repr_html',False)
pd.set_option('display.max_columns',10)
pd.set_option('display.max_rows',10)
Explanation: Pandas
Pandas is a high-performance python library that provides a comprehensive set of data structures for manipulating tabular data, providing high-performance indexing, automatic alignment, reshaping, grouping, joining and statistical analyses capabilities.
The two primary data structures in pandas are the Series and the DataFrame objects.
Series Object
The Series object is the fundamental building block of pandas. A Series represents an one-dimensional array based on the NumPy ndarray but with a labeled index that significantly helps to access the elements.
A Series always has an index even if one is not specified, by default pandas will create an index that consists of sequential integers starting from zero. Access to elements is not by integer position but using values in the index referred as Labels.
Importing pandas into the application is simple. It is common to import both pandas and numpy with their objects mapped into the pd and np namespaces respectively.
End of explanation
# create one item series
s1 = pd.Series(1)
s1
Explanation: Creating Series
A Series can be created and initialised by passing either a scalar value, a NumPy nd array, a Python list or a Python Dict as the data parameter of the Series constructor.
End of explanation
# get value with label 0
s1[0]
# create from list
s2 = pd.Series([1,2,3,4,5])
s2
# get the values in the series
s2.values
# get the index of the series
s2.index
Explanation: '0' is the index and '1' is the value. The data type (dtype) is also shown. We can also retrieve the value using the associated index.
End of explanation
# explicitly create an index
# index is alpha, not an integer
s3 = pd.Series([1,2,3], index=['a','b','c'])
s3
s3.index
Explanation: Creating Series with named index
Pandas will create different index types based on the type of data identified in the index parameter. These different index types are optimized to perform indexing operations for that specific data type. To specify the index at the time of creation of the Series, use the index parameter of the constructor.
End of explanation
# look up by label value and not object position
s3['b']
# position also works
s3[2]
# create series from an existing index
# scalar value will be copied at each index label
s4 = pd.Series(2,index=s2.index)
s4
Explanation: Please note the type of the index items. It is not string but 'object'.
End of explanation
np.random.seed(123456)
pd.Series(np.random.randn(5))
# 0 through 9
pd.Series(np.linspace(0,9,10))
# o through 8
pd.Series(np.arange(0,9))
Explanation: It is a common practice to initialize the Series objects using NumPy ndarrays, and with various NumPy functions that create arrays. The following code creates a Series from five normally distributed values:
End of explanation
s6 = pd.Series({'a':1,'b':2,'c':3,'d':4})
s6
Explanation: A Series can also be created from a Python dictionary. The keys of the dictionary are used as the index lables for the Series:
End of explanation
# example series which also contains a NaN
s = pd.Series([0,1,1,2,3,4,5,6,7,np.NaN])
s
# length of the Series
len(s)
s.size
# shape is a tuple with one value
s.shape
# number of values not part of NaN can be found using count() method
s.count()
# all unique values
s.unique()
# count of non-NaN values, returned max to min order
s.value_counts()
Explanation: Size, Shape, Count and Uniqueness of Values
End of explanation
# first five
s.head()
# first three
s.head(3)
# last five
s.tail()
# last 2
s.tail(n=2) # equivalent to s.tail(2)
Explanation: Peeking at data with heads, tails and take
pandas provides the .head() and .tail() methods to examine just the first few or last records in a Series. By default, these return the first five or last rows respectively, but you can use the n parameter or just pass an integer to specify the number of rows:
End of explanation
# only take specific items
s.take([0,3,9])
Explanation: The .take() method will return the rows in a series that correspond to the zero-based positions specified in a list:
End of explanation
# single item lookup
s3['a']
# lookup by position since index is not an integer
s3[2]
# multiple items
s3[['a','c']]
# series with an integer index but not starting with 0
s5 = pd.Series([1,2,3], index =[11,12,13])
s5[12]  # by value as value passed and index are both integer
Explanation: Looking up values in Series
Values in a Series object can be retrieved using the [] operator and passing either a single index label or a list of index labels.
End of explanation
# force lookup by index label
s5.loc[12]
Explanation: To alleviate the potential confusion in determining the label-based lookups versus position-based lookups, index based lookup can be enforced using the .loc[] accessor:
End of explanation
# force lookup by position or location
s5.iloc[1]
# multiple items by index label
s5.loc[[12,10]]
# multiple items by position or location
s5.iloc[[1,2]]
Explanation: Lookup by position can be enforced using the iloc[] accessor:
End of explanation
s5.loc[[12,-1,15]]
Explanation: If a location / position passed to .iloc[] in a list is out of bounds, an exception will be thrown. This is different than with .loc[], which if passed a label that does not exist, will return NaN as the value for that label:
End of explanation
s3
# label based lookup
s3.ix[['a','b']]
# position based lookup
s3.ix[[1,2]]
Explanation: A Series also has a property .ix that can be used to look up items either by label or by zero-based array position.
End of explanation
# this looks by label and not position
# note that 1,2 have NaN as those labels do not exist in the index
s5.ix[[1,2,10,11]]
Explanation: This can become complicated if the indexes are integers and you pass a list of integers to ix. Since they are of the same type, the lookup will be by index label instead of position:
End of explanation
s6 = pd.Series([1,2,3,4], index=['a','b','c','d'])
s6
s7 = pd.Series([4,3,2,1], index=['d','c','b','a'])
s7
s6 + s7
Explanation: Alignment via index labels
A fundamental difference between a NumPy ndarray and a pandas Series is the ability of a Series to automatically align data from another Series based on label values before performing an operation.
End of explanation
a1 = np.array([1,2,3,4,5])
a2 = np.array([5,4,3,2,1])
a1 + a2
Explanation: This is a very different result that what it would have been if it were two pure NumPy arrays being added. A NumPy ndarray would add the items in identical positions of each array resulting in different values.
End of explanation
# multiply all values in s3 by 2
s3 * 2
# scalar series using the s3's index
# not efficient as it will no use vectorisation
t = pd.Series(2,s3.index)
s3 * t
Explanation: The process of adding two Series objects differs from the process of addition of arrays as it first aligns data based on index label values instead of simply applying the operation to elements in the same position. This becomes significantly powerful when using pandas Series to combine data based on labels instead of having to first order the data manually.
Arithmetic Operations
Arithemetic Operations <pre>(+,-,*,/)</pre> can be applied either to a Series or between 2 Series objects
End of explanation
# we will add this to s9
s8 = pd.Series({'a':1,'b':2,'c':3,'d':5})
s8
s9 = pd.Series({'b':6,'c':7,'d':9,'e':10})
s9
# NaN's result for a and e demonstrates alignment
s8 + s9
s10 = pd.Series([1.0,2.0,3.0],index=['a','a','b'])
s10
s11 = pd.Series([4.0,5.0,6.0], index=['a','a','c'])
s11
# will result in four 'a' index labels
s10 + s11
Explanation: To reinforce the point that alignment is being performed when applying arithmetic operations across two Series objects, look at the following two Series as examples:
End of explanation
nda = np.array([1,2,3,4,5])
nda.mean()
# mean of numpy array values with a NaN
nda = np.array([1,2,3,4,np.NaN])
nda.mean()
# Series object ignores NaN values - does not get factored
s = pd.Series(nda)
s.mean()
# handle NaN values like Numpy
s.mean(skipna=False)
Explanation: The reason for the above result is that during alignment, pandas actually performs a cartesian product of the sets of all the unique index labels in both Series objects, and then applies the specified operation on all items in the products.
To explain why there are four 'a' index values s10 contains two 'a' labels and s11 also contains two 'a' labels. Every combination of 'a' labels in each will be calculated resulting in four 'a' labels. There is one 'b' label from s10 and one 'c' label from s11. Since there is no matching label for either in the other Series object, they only result in a sing row in the resulting Series object.
Each combination of values for 'a' in both Series are computed, resulting in the four values: 1+4,1+5,2+4 and 2+5.
So remember that an index can have duplicate labels, and during alignment this will result in a number of index labels equivalent to the products of the number of the labels in each Series.
The special case of Not-A-Number (NaN)
pandas mathematical operators and functions handle NaN in a special manner (compared to NumPy ndarray) that does not break the computations. pandas is lenient with missing data assuming that it is a common situation.
End of explanation
# which rows have values that are > 5
s = pd.Series(np.arange(0,10))
s > 5
# select rows where values are > 5
# overloading the Series object [] operator
logicalResults = s > 5
s[logicalResults]
# a little shorter version
s[s > 5]
# using & operator
s[(s>5)&(s<9)]
# using | operator
s[(s > 3) | (s < 5)]
# are all items >= 0?
(s >=0).all()
# are any items < 2
s[s < 2].any()
Explanation: Boolean selection
Items in a Series can be selected, based on the value instead of index labels, via the utilization of a Boolean selection.
End of explanation
(s < 2).sum()
Explanation: The result of these logical expressions is a Boolean selection, a Series of True and False values. The .sum() method of a Series, when given a series of Boolean values, will treat True as 1 and False as 0. The following demonstrates using this to determine the number of items in a Series that satisfy a given expression:
End of explanation
# sample series of five items
s = pd.Series(np.random.randn(5))
s
# change the index
s.index = ['a','b','c','d','e']
s
# concat copies index values verbatim
# potentially making duplicates
np.random.seed(123456)
s1 = pd.Series(np.random.randn(3))
s2 = pd.Series(np.random.randn(3))
combined = pd.concat([s1,s2])
combined
# reset the index
combined.index = np.arange(0,len(combined))
combined
Explanation: Reindexing a Series
Reindexing in pandas is a process that makes the data in a Series or DataFrame match a given set of labels.
This process of performing a reindex includes the following steps:
1. Reordering existing data to match a set of labels.
2. Inserting NaN markers where no data exists for a label.
3. Possibly, filling missing data for a label using some type of logic
End of explanation
np.random.seed(123456)
s1 = pd.Series(np.random.randn(4),['a','b','c','d'])
# reindex with different number of labels
# results in dropped rows and/or NaN's
s2 = s1.reindex(['a','c','g'])
s2
Explanation: Greater flexibility in creating a new index is provided using the .reindex() method. An example of the flexibility of .reindex() over assigning the .index property directly is that the list provided to .reindex() can be of a different length than the number of rows in the Series:
End of explanation
# s2 is a different series than s1
s2['a'] = 0
s2
# this did not modify s1
s1
Explanation: There are several things here that are important to point out about .reindex() method.
First is that the result of .reindex() method is a new Series. This new Series has an index with labels that are provided as parameter to reindex().
For each item in the given parameter list, if the original Series contains that label, then the value is assigned to that label.
If that label does not exist in the original Series, pandas assigns a NaN value.
Rows in the Series without a label specified in the parameter of .reindex() is not included in the result.
To demonstrate that the result of .reindex() is a new Series object, changing a value in s2 does not change the values in s1:
End of explanation
# different types for the same values of labels causes big issue
s1 = pd.Series([0,1,2],index=[0,1,2])
s2 = pd.Series([3,4,5],index=['0','1','2'])
s1 + s2
Explanation: Reindex is also useful when you want to align two Series to perform an operation on matching elements from each series; however, for some reason, the two Series has index labels that will not initially align.
End of explanation
# reindex by casting the label types and we will get the desired result
s2.index = s2.index.values.astype(int)
s1 + s2
Explanation: The reason why this happens in pandas are as follows:
pandas first tries to align by the indexes and finds no matches, so it copies the index labels from the first series and tries to append the indexes from the second Series.
However, since they are different type, it defaults back to zero-based integer sequence that results in duplicate values.
Finally, all values are NaN because the operation tries to add the item in the first Series with the integer label 0, which has a value of 0, but can't find the item in the other series and therefore the result in NaN.
End of explanation
# fill with 0 instead on NaN
s2 = s.copy()
s2.reindex(['a','f'],fill_value=0)
Explanation: The default action of inserting NaN as a missing value during reindexing can be changed by using the fill_value parameter of the method.
End of explanation
# create example to demonstrate fills
s3 = pd.Series(['red','green','blue'],index=[0,3,5])
s3
# forward fill using ffill method
s3.reindex(np.arange(0,7), method='ffill')
# backward fill using bfill method
s3.reindex(np.arange(0,7),method='bfill')
Explanation: When performing a reindex on ordered data such as a time series, it is possible to perform interpolation or filling of values. The following example demonstrates forward filling, often referred to as "last known value".
End of explanation
np.random.seed(123456)
s = pd.Series(np.random.randn(3),index=['a','b','c'])
s
# change a value in the Series
# this done in-place
# a new Series is not returned that has a modified value
s['d'] = 100
s
# value at a specific index label can be changed by assignment:
s['d'] = -100
s
Explanation: Modifying a Series in-place
There are several ways that an existing Series can be modified in-place having either its values changed or having rows added or deleted.
A new item can be added to a Series by assigning a value to an index label that does not already exist.
End of explanation
del(s['a'])
s
Explanation: Items can be removed from a Series using the del() function and passing the index label(s) to be removed.
End of explanation
# a series to use for slicing
# using index labels not starting at 0 to demonstrate
# position based slicing
s = pd.Series(np.arange(100,110),index=np.arange(10,20))
s
# items at position 0,2,4
s[0:6:2]
# equivalent to
s.iloc[[0,2,4]]
# first five by slicing, same as .head(5)
s[:5]
# fourth position to the end
s[4:]
# every other item in the first five positions
s[:5:2]
# every other item starting at the fourth position
s[4::2]
# reverse the series
s[::-1]
# every other starting at position 4, in reverse
s[4::-2]
# :-2 which means positions 0 through (10-2) which is [8]
s[:-2]
# last 3 items
# equivalent to tail(3)
s[-3:]
# equivalent to s.tail(4).head(3)
s[-4:-1]
Explanation: Slicing a Series
End of explanation
# preserve s
# slice with first 2 rows
copy = s.copy()
slice = copy[:2]
slice
Explanation: An important thing to keep in mind when using slicing, is that the result of the slice is actually a view into the original Series. Modification of values through the result of the slice will modify the original Series.
End of explanation
slice[11] = 1000
copy
Explanation: Now the assignment of a value to an element of a slice will change the value in the original Series:
End of explanation
# used to demonstrate the next two slices
s = pd.Series(np.arange(0,5),index=['a','b','c','d','e'])
s
# slicing with integer values will extract items based on position:
s[1:3]
# with non-integer index, it is also possible to slice with values in the same type of the index:
s['b':'d']
Explanation: Slicing can be performed on Series objects with a non-integer index.
End of explanation
<END_TASK>