<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 Pandas Crash Course
<img style="float
Step1:  Pandas Series and DataFrame objects
There are two main data structures in pandas
Step2:  Dataframes use the IPython display method to look pretty, but will show just fine when printed also. (There's a way to make all of the dataframes print pretty via the IPython.display.display method, but this isn't necessary to view the values)
Step3:  Indices can be named
Step4:  Data Input Output
Step5:  CSV Files
Step6:  hdf5 files
Step7:  Data types
Show the datatypes of each column
Step8:  We can create dataframes of multiple datatypes
Step9:  We can also set the 'cow color' column to a category
Step10:  Indexing and Setting Data
Pandas does a lot of different operations, here are the meat and potatoes. The following describes the indexing of data, but setting the data is as simple as a reassignment.
Step11:  Head and Tail
Print the beginning and ending entries of a pandas data structure
Step12:  We can also separate the metadata (labels, etc) from the data, yielding a numpy-like output.
Step13:  Indexing Data
Pandas provides the means to index data via named columns, or as numpy like indices. Indexing is [row, column], just as it was in numpy.
Data is visible via column
Step14:  Note that tab completion is enabled for column names
Step15:  <div>
<img style="float
Step16:  Label based indexing (.loc)
Slice based on the labels.
Step17:  Integer based indexing (.iloc)
Slice based on the index number.
Step18:  Fast single element label indexing (.at) - fast .loc
Intended for fast, single indexes.
Step19:  Fast single element label indexing (.iat) - fast .iloc
Intended for fast, single indexes.
Step20:  Logical indexing
A condition is used to select the values within a slice or the entire Pandas object. Using a conditional statement, a true/false DataFrame is produced
Step21:  That matrix can then be used to index the DataFrame
Step22:  Logical indexing via isin
It's also possible to filter via the index value
Step23:  Sorting by column
Step24:  Dealing with Missing Data
By convention, pandas uses the NaN value to represent missing data. There are a few functions surrounding the handling of NaN values
Step25:  Print a logical DataFrame where NaN is located
Step26:  Drop all rows with NaN
Step27:  Replace NaN entries
Step28:  Concatenating and Merging Data
Bringing together DataFrames or Series objects
Step29:  Append
Adding an additional group after the first group
Step30:  SQL-like merging
Pandas can do structured query language (SQL) like merges of data
Step31:  Grouping Operations
Often, there is a need to summarize the data or change the output of the data to make it easier to work with, especially for categorical data types.
Step32:  Pivot Table
Without changing the data in any way, summarize the output in a different format. Specify the indicies, columns, and values
Step33:  Stacking
Column labels can be brought into the rows.
Step34:  Groupby
Groupby groups values, creating a Python object to which functions can be applied
Step35:  Operations on Pandas Data Objects
Wether it's the entire data frame or a series within a single dataframe, there are a variety of methods that can be applied. Here's a list of a few helpful ones
Step36:  Rotation
Note that the values rotated out leave NaN behind
Step37:  Add, subtract, multiply, divide
Step38:  Histogram
Step39:  Describe
Excluding NaN values, print some descriptive statistics about the collection of values.
Step40:  Transpose
Exchange the rows and columns (flip about the diagonal)
Step41:  Applying Any Function to Pandas Data Objects
Pandas objects have methods that allow function to be applied with greater control, namely the .apply function
Step42:  Lambda functions may also be used
Step43:  String functions
Step44:  Plotting
Pandas exposes the matplotlib library for use.
Step45:  Matplotlib styles are available too
<ASSISTANT_TASK:>
Python Code:
# Python imports
import pandas as pd
import numpy as np
import matplotlib as mpl
import matplotlib.pyplot as plt
Explanation: Pandas Crash Course
<img style="float: right; border:3px solid black" src="images/10_Panda_DailyMail_7_Nov_2013.jpg" border="5" width=30%>
Pandas is a Python package that aims to make working with data as easy and intuitive as possible. It fills the role of a foundational real world data manipulation library and interfaces with many other Python packages.
By the end of this file you should have seen simple examples of:
Pandas Series and DataFrame objects
Data IO
Data types
Indexing and setting data
Dealing with missing data
Concatinating and merging data
Grouping Operations
Operations on Pandas data objects
Applying any function to Pandas data objects
Plotting
Further Reading:
http://pandas.pydata.org/pandas-docs/stable/10min.html
https://pandas.pydata.org/pandas-docs/stable/comparison_with_sql.html#compare-with-sql-join  
Image Credit: David Jenkins at Bifengxia Panda Reserve in Chengdu
End of explanation
series1 = pd.Series([1,2,3,4])
print(series1)
Explanation: Pandas Series and DataFrame objects
There are two main data structures in pandas:
-  Series (1 dimensional data)
-  Dataframes (2 dimensional data)
-  There are other, lesser used data structures used for higher dimensional data, but are less frequently used
    - Panel (3 dimensional data) - panel will be removed from future versions of Pandas and replaced with xarray
    -  Xarray (>2 dimensions)
Here, the 1- and 2-dimensional data sets are the focus of this lesson.
Pandas DataFrames are analogus to R's data.frame, but aim to provide additional functionality. 
Both dataframes and series data structures have indicies, which are shown on the left:
End of explanation
df1 = pd.DataFrame([[1,2,3,4],[10,20,30,40]])
print(df1)
df1
Explanation: Dataframes use the IPython display method to look pretty, but will show just fine when printed also. (There's a way to make all of the dataframes print pretty via the IPython.display.display method, but this isn't necessary to view the values):
End of explanation
# Rename the columns
df1.columns = ['A','B','C','D']
df1.index = ['zero','one']
df1
# Create the dataframe with the columns
df1 = pd.DataFrame([[1,2,3,4],[10,20,30,40]], columns=['A','B','C',"D"], index=['zero','one'])
df1
Explanation: Indices can be named:
End of explanation
df1 = pd.DataFrame(np.random.randn(5,4), columns = ['A','B','C','D'], index=['zero','one','two','three','four'])
print(df1)
Explanation: Data Input Output
End of explanation
df1.to_csv('datafiles/pandas_df1.csv')
!ls datafiles
df2 = pd.read_csv('datafiles/pandas_df1.csv', index_col=0)
print(df2)
Explanation: CSV Files
End of explanation
df1.to_hdf('datafiles/pandas_df1.h5', 'df')
!ls datafiles
df2 = pd.read_hdf('datafiles/pandas_df1.h5', 'df')
print(df2)
Explanation: hdf5 files
End of explanation
df2.dtypes
Explanation: Data types
Show the datatypes of each column:
End of explanation
col1 = range(6)
col2 = np.random.rand(6)
col3 = ['zero','one','two','three','four','five']
col4 = ['blue', 'cow','blue', 'cow','blue', 'cow']
df_types = pd.DataFrame( {'integers': col1, 'floats': col2, 'words': col3, 'cow color': col4} )
print(df_types)
df_types.dtypes
Explanation: We can create dataframes of multiple datatypes:
End of explanation
df_types['cow color'] = df_types['cow color'].astype("category")
df_types.dtypes
Explanation: We can also set the 'cow color' column to a category:
End of explanation
time_stamps = pd.date_range(start='2000-01-01', end='2000-01-20', freq='D') # Define index of time stamps
df1 = pd.DataFrame(np.random.randn(20,4), columns = ['A','B','C','D'], index=time_stamps)
print(df1)
Explanation: Indexing and Setting Data
Pandas does a lot of different operations, here are the meat and potatoes. The following describes the indexing of data, but setting the data is as simple as a reassignment.
End of explanation
df1.head(3) # Show the first n rows, default is 5
df1.tail() # Show the last n rows
Explanation: Head and Tail
Print the beginning and ending entries of a pandas data structure
End of explanation
df1.columns
df1.values
Explanation: We can also separate the metadata (labels, etc) from the data, yielding a numpy-like output.
End of explanation
df1['A'].head() # df1.A.head() is equivalent
Explanation: Indexing Data
Pandas provides the means to index data via named columns, or as numpy like indices. Indexing is [row, column], just as it was in numpy.
Data is visible via column:
End of explanation
df1.A
Explanation: Note that tab completion is enabled for column names:
End of explanation
df1[:2]
Explanation: <div>
<img style="float: left;" src="images/10-01_column-tab.png" width=30%>
</div>
We can specify row ranges:
End of explanation
df1.loc[:'2000-01-5',"A"] # Note that this includes the upper index
Explanation: Label based indexing (.loc)
Slice based on the labels.
End of explanation
df1.iloc[:3,0] # Note that this does not include the upper index like numpy
Explanation: Integer based indexing (.iloc)
Slice based on the index number.
End of explanation
index_timestamp = pd.Timestamp('2000-01-03') # Create a timestamp object to index
df1.at[index_timestamp,"A"]  # Index using timestamp (vs string)
Explanation: Fast single element label indexing (.at) - fast .loc
Intended for fast, single indexes.
End of explanation
df1.iat[3,0]
Explanation: Fast single element label indexing (.iat) - fast .iloc
Intended for fast, single indexes.
End of explanation
df1.head()>0.5
Explanation: Logical indexing
A condition is used to select the values within a slice or the entire Pandas object. Using a conditional statement, a true/false DataFrame is produced:
End of explanation
df1[df1>0.5].head() # Note that the values that were 'False' are 'NaN'
Explanation: That matrix can then be used to index the DataFrame:
End of explanation
df_types
bool_series = df_types['cow color'].isin(['blue'])
print(bool_series) # Show the logical indexing
df_types[bool_series] # Index where the values are true
Explanation: Logical indexing via isin
It's also possible to filter via the index value:
End of explanation
df_types.sort_values(by="floats")
Explanation: Sorting by column
End of explanation
df_nan = pd.DataFrame(np.random.rand(6,2), columns = ['A','B'])
df_nan
df_nan['B'] = df_nan[df_nan['B']>0.5] # Prints NaN Where ['B'] <= 0.5
print(df_nan)
Explanation: Dealing with Missing Data
By convention, pandas uses the NaN value to represent missing data. There are a few functions surrounding the handling of NaN values:
End of explanation
df_nan.isnull()
Explanation: Print a logical DataFrame where NaN is located:
End of explanation
df_nan.dropna(how = 'any')
Explanation: Drop all rows with NaN:
End of explanation
df_nan.fillna(value = -1)
Explanation: Replace NaN entries:
End of explanation
df1 = pd.DataFrame(np.zeros([3,3], dtype=np.int))
df1
df2 = pd.concat([df1, df1], axis=0)
df2 = df2.reset_index(drop=True) # Renumber indexing
df2
Explanation: Concatenating and Merging Data
Bringing together DataFrames or Series objects:
Concatenate
End of explanation
newdf = pd.DataFrame({0: [1], 1:[1], 2:[1]})
print(newdf)
df3 = df2.append(newdf, ignore_index=True)
df3
Explanation: Append
Adding an additional group after the first group:
End of explanation
left = pd.DataFrame({'numbers': ['K0', 'K1', 'K2', 'K3'],
    'English': ['one', 'two', 'three', 'four'],
    'Spanish': ['uno', 'dos', 'tres', 'quatro'],
    'German': ['erste', 'zweite','dritte','vierte']})
left
right = pd.DataFrame({'numbers': ['K0', 'K1', 'K2', 'K3'],
    'French': ['un', 'deux', 'trois', 'quatre'],
    'Afrikaans': ['een', 'twee', 'drie', 'vier']})
right
result = pd.merge(left, right, on='numbers')
result
Explanation: SQL-like merging
Pandas can do structured query language (SQL) like merges of data:
End of explanation
dfg = pd.DataFrame({'A': ['clogs','sandals','jellies']*2,
                    'B': ['socks','footies']*3,
                    'C': [1,1,1,3,2,2],
                    'D': np.random.rand(6)})
dfg
Explanation: Grouping Operations
Often, there is a need to summarize the data or change the output of the data to make it easier to work with, especially for categorical data types.
End of explanation
dfg.pivot_table(index=['A','B'], columns=['C'], values='D')
Explanation: Pivot Table
Without changing the data in any way, summarize the output in a different format. Specify the indicies, columns, and values:
End of explanation
dfg.stack()
Explanation: Stacking
Column labels can be brought into the rows.
End of explanation
dfg.groupby(['B']).count()
dfg.groupby(['A']).mean()
Explanation: Groupby
Groupby groups values, creating a Python object to which functions can be applied:
End of explanation
dfg['D'].mean()
Explanation: Operations on Pandas Data Objects
Wether it's the entire data frame or a series within a single dataframe, there are a variety of methods that can be applied. Here's a list of a few helpful ones:
Simple statistics (mean, stdev, etc).
End of explanation
dfg['D']
dfg_Ds = dfg['D'].shift(2)
dfg_Ds 
Explanation: Rotation
Note that the values rotated out leave NaN behind:
End of explanation
dfg['D'].div(dfg_Ds )
Explanation: Add, subtract, multiply, divide:
Operations are element-wise:
End of explanation
dfg
dfg['C'].value_counts()
Explanation: Histogram
End of explanation
df_types.describe()
Explanation: Describe
Excluding NaN values, print some descriptive statistics about the collection of values.
End of explanation
df_types.T
Explanation: Transpose
Exchange the rows and columns (flip about the diagonal):
End of explanation
def f(x): # Define function
    return x + 1
dfg['C'].apply(f)
Explanation: Applying Any Function to Pandas Data Objects
Pandas objects have methods that allow function to be applied with greater control, namely the .apply function:
End of explanation
dfg['C'].apply(lambda x: x + 1)
Explanation: Lambda functions may also be used
End of explanation
dfg['A'].str.title() # Make the first letter uppercase
Explanation: String functions:
Pandas has access to string methods:
End of explanation
n = 100
X = np.linspace(0, 5, n)
Y1,Y2 = np.log((X)**2+2), np.sin(X)+2
dfp = pd.DataFrame({'X' : X, 'Y1': Y1, 'Y2': Y2})
dfp.head()
dfp.plot(x = 'X')
plt.show() 
Explanation: Plotting
Pandas exposes the matplotlib library for use.
End of explanation
style_name = 'classic'
plt.style.use(style_name)
dfp.plot(x = 'X')
plt.title('Log($x^2$) and Sine', fontsize=16)
plt.xlabel('X Label', fontsize=16)
plt.ylabel('Y Label', fontsize=16)
plt.show() 
mpl.rcdefaults() # Reset matplotlib rc defaults
Explanation: Matplotlib styles are available too:
End of explanation
<END_TASK>