<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 Python/Pandas Refresher
Overview of today's topics
Step1:  1. Quick Python Refresher
A quick overview of ubiquitous programming concepts including data types, for loops, if-then-else conditionals, and functions.
Step2:  2. pandas Series and DataFrames
pandas has two primary data structures we will work with
Step3:  2b. pandas DataFrames
Step4:  3. Loading data
In practice, you'll work with data by loading a dataset file into pandas. CSV is the most common format. But pandas can also ingest tab-separated data, JSON, and proprietary file formats like Excel .xlsx files, Stata, SAS, and SPSS.
Below, notice what pandas's read_csv function does
Step5:  4. Selecting and slicing data from a DataFrame
Step6:  4a. Select DataFrame's column(s) by name
Step7:  4b. Select row(s) by label
Step8:  4c. Select by (integer) position
Step9:  4d. Select/filter by value
You can subset or filter a dataframe for based on the values in its rows/columns.
Step10:  5. Merge and concatenate
5a. Merging DataFrames
Merging joins data together, aligned by the index. Assume here that we had two separate data sets, with unique uc_id identifiers.
Step11:  5b. Concatenating DataFrames
Concatenating smushes data together along some axis
Step12:  6. Grouping and summarizing
Step13:  7. Vectorization, Map, and Apply
Avoid slow iteration in pandas! Use vectorization (best) or map/apply (ok) when possible. These methods apply a function across elements, rows, and columns of a pandas DataFrame or Series. But they have some important and often confusing differences. Some tips
Step14:  .map() applies a function element-wise on a series
Step15:  A lambda function is a simple, one-off, anonymous function. You can't call it again later because it doesn't have a name. It just lets you repeatedly perform some operation across a series of values (in our case, a column in our dataframe) using a minimal amount of code.
Step16:  .apply() is like .map(), but it works on a row or column basis on an entire DataFrame (specify the axis)
Step17:  .applymap() works element-wise on an entire DataFrame. This is like doing a .map() to each column in the DataFrame.
Step18:  8. Hierarchical Indexing
A pandas index can comprise multiple values. It's very powerful, but gets tricky sometimes. See the docs.
In general, it's a good idea to sort your index for better querying performance later.
Step19:  So what's the benefit of multi-indexing like this? Your dataframe becomes a very fast look-up table. A good index is one that gives you a useful handle to select rows.
Step20:  Stacking, unstacking, and pivoting are common ways to reshape data with a MultiIndex. Depending on the form your original data come in, they may be useful in the data prep phase to make your data easier to analyze. We'll discuss data prep, cleaning, and (review) visualization next week.
<ASSISTANT_TASK:>
Python Code:
import numpy as np
import pandas as pd
Explanation: Python/Pandas Refresher
Overview of today's topics:
Quick Python refresher
pandas overview
Load data files
Select, filter, and slice data from a dataset
Merging and concatenating datasets
Grouping and summarizing data
Vectorization, map, and apply
Hierarchical indexing
End of explanation
# integers (int)
x = 100
type(x)
# floating-point numbers (float)
x = 100.5
type(x)
# sequence of characters (str)
x = 'Los Angeles, CA 90089'
len(x)
# list of items
x = [1, 2, 3, 'USC']
len(x)
# sets are unique
x = {2, 2, 3, 3, 1}
x
# tuples are immutable sequences
latlng = (34.019425, -118.283413)
type(latlng)
# you can unpack a tuple
lat, lng = latlng
type(lat)
# dictionary of key:value pairs
person = {'first_name': 'Geoff', 'last_name': 'Boeing', 'employer': 'USC'}
type(person)
# you can convert types
x = '100'
print(type(x))
y = int(x)
print(type(y))
# you can loop through an iterable, such as a list or tuple
for coord in latlng:
    print('Current coordinate is:', coord)
# loop through a dictionary keys and values as tuples
for key, value in person.items():
    print(key, value)
# booleans are trues/falses
x = 101
x > 100
# use two == for equality and one = for assignment
x == 100
# if, elif, else for conditional branching execution
x = 101
if x > 100:
    print('Value is greater than 100.')
elif x < 100:
    print('Value is less than 100.')
else:
    print('Value is 100.')
# use functions to encapsulate and reuse bits of code
def convert_items(my_list, new_type=str):
    # convert each item in a list to a new type
    new_list = [new_type(item) for item in my_list]
    return new_list
l = [1, 2, 3, 4]
convert_items(l)
Explanation: 1. Quick Python Refresher
A quick overview of ubiquitous programming concepts including data types, for loops, if-then-else conditionals, and functions.
End of explanation
# a pandas series is based on a numpy array: it's fast, compact, and has more functionality
# perhaps most notably, it has an index which allows you to work naturally with tabular data
my_list = [8, 5, 77, 2]
my_series = pd.Series(my_list)
my_series
# look at a list-representation of the index
my_series.index.tolist()
# look at the series' values themselves
my_series.values
# what's the data type of the series' values?
type(my_series.values)
# what's the data type of the individual values themselves?
my_series.dtype
Explanation: 2. pandas Series and DataFrames
pandas has two primary data structures we will work with: Series and DataFrames
2a. pandas Series
End of explanation
# a dict can contain multiple lists and label them
my_dict = {'hh_income'  : [75125, 22075, 31950, 115400],
           'home_value' : [525000, 275000, 395000, 985000]}
my_dict
# a pandas dataframe can contain one or more columns
# each column is a pandas series
# each row is a pandas series
# you can create a dataframe by passing in a list, array, series, or dict
df = pd.DataFrame(my_dict)
df
# the row labels in the index are accessed by the .index attribute of the DataFrame object
df.index.tolist()
# the column labels are accessed by the .columns attribute of the DataFrame object
df.columns
# the data values are accessed by the .values attribute of the DataFrame object
# this is a numpy (two-dimensional) array
df.values
Explanation: 2b. pandas DataFrames
End of explanation
# load a data file
# note the relative filepath! where is this file located?
# use dtype argument if you don't want pandas to guess your data types
df = pd.read_csv('../../data/world_cities.csv')
# dataframe shape as rows, columns
df.shape
# or use len to just see the number of rows
len(df)
# view the dataframe's "head"
df.head()
# view the dataframe's "tail"
df.tail()
Explanation: 3. Loading data
In practice, you'll work with data by loading a dataset file into pandas. CSV is the most common format. But pandas can also ingest tab-separated data, JSON, and proprietary file formats like Excel .xlsx files, Stata, SAS, and SPSS.
Below, notice what pandas's read_csv function does:
recognize the header row and get its variable names
read all the rows and construct a pandas DataFrame (an assembly of pandas Series rows and columns)
construct a unique index, beginning with zero
infer the data type of each variable (ie, column)
End of explanation
# CHEAT SHEET OF COMMON TASKS
# Operation                       Syntax           Result
#------------------------------------------------------------
# Select column by name           df[col]          Series
# Select columns by name          df[col_list]     DataFrame
# Select row by label             df.loc[label]    Series
# Select row by integer location  df.iloc[loc]     Series
# Slice rows by label             df.loc[a:c]      DataFrame
# Select rows by boolean vector   df[mask]         DataFrame
Explanation: 4. Selecting and slicing data from a DataFrame
End of explanation
# select a single column by column name
# this is a pandas series
df['resident_pop']
# select multiple columns by a list of column names
# this is a pandas dataframe that is a subset of the original
df[['resident_pop', 'built_up_area']]
# create a new column by assigning df['new_col'] to some values
df['pop_density'] = df['resident_pop'] / df['built_up_area']
# you can do vectorized math operations on any numeric columns
df['pop_density_1000s'] = df['pop_density'] / 1000
# inspect the results
df[['resident_pop', 'built_up_area', 'pop_density', 'pop_density_1000s']].head()
Explanation: 4a. Select DataFrame's column(s) by name
End of explanation
# use .loc to select by row label
# returns the row as a series whose index is the dataframe column names
df.loc[0]
# use .loc to select single value by row label, column name
df.loc[0, 'resident_pop']
# slice of rows from label 5 to label 7, inclusive
# this returns a pandas dataframe
df.loc[5:7]
# slice of rows from label 5 to label 7, inclusive
# slice of columns from uc_names to world_subregion, inclusive
df.loc[1:3, 'uc_names':'world_subregion']
# subset of rows from with labels in list
# subset of columns with names in list
df.loc[[1, 3], ['uc_names', 'world_subregion']]
# you can use a column of identifiers as the index (indices do not *need* to be unique)
# uc_id values uniquely identify each row (but verify!)
df = df.set_index('uc_id')
df.index.is_unique
df.head()
# .loc works by label, not by position in the dataframe
try:
    df.loc[0]
except KeyError as e:
    print('label not found')
# the index now contains uc_id values, so you have to use .loc accordingly to select by row label
df.loc[14]
Explanation: 4b. Select row(s) by label
End of explanation
# get the row in the zero-th position in the dataframe
df.iloc[0]
# you can slice as well
# note, while .loc is inclusive, .iloc is not
# get the rows from position 0 up to but not including position 3 (ie, rows 0, 1, and 2)
df.iloc[0:3]
# get the value from the row in position 3 and the column in position 2 (zero-indexed)
df.iloc[3, 2]
Explanation: 4c. Select by (integer) position
End of explanation
# filter the dataframe by urban areas with more than 25 million residents
df[df['resident_pop'] > 25000000]
# what exactly did that do? let's break it out.
df['resident_pop'] > 25000000
# essentially a true/false mask that filters by value
mask = df['resident_pop'] > 25000000
df[mask]
# you can chain multiple conditions together
# pandas logical operators are: | for or, & for and, ~ for not
# these must be grouped by using parentheses due to order of operations
mask = (df['resident_pop'] > 25000000) & (df['built_up_area'] > 2000)
df[mask]
# which urban areas have 25 million residents and either 2000 km2 area or >200 meter avg elevation?
mask1 = df['resident_pop'] > 25000000
mask2 = df['built_up_area'] > 2000
mask3 = df['avg_elevation'] > 200
mask = mask1 & (mask2 | mask3)
df[mask]
# see the mask
mask
# ~ means not... it essentially flips trues to falses and vice-versa
~mask
# which urban areas contain a city with "New " in its name?
mask = df['uc_names'].str.contains('New ')
df.loc[mask, ['uc_names', 'country']]
# now it's your turn
# create a new subset dataframe containing all urban areas in the US with >5 million residents
# how many rows did you get?
Explanation: 4d. Select/filter by value
You can subset or filter a dataframe for based on the values in its rows/columns.
End of explanation
# create a subset dataframe with climate related variables
climate_cols = ['core_city', 'avg_elevation', 'avg_precipitation', 'avg_temperature', 'climate_classes']
df_climate = df[climate_cols].sample(2000).sort_values('avg_temperature', ascending=True)
df_climate.head()
# create a subset dataframe with economic variables
econ_cols = ['core_city', 'gdp_ppp', 'night_light_em', 'un_income_class']
df_econ = df[econ_cols].sample(2000).sort_values('gdp_ppp', ascending=False)
df_econ.head()
# merge them together, aligning rows based on their labels in the index
df_merged = pd.merge(left=df_econ, right=df_climate, how='inner', left_index=True, right_index=True)
df_merged.head()
# reset df_econ's index
df_econ = df_econ.reset_index()
df_econ.head()
# merge them together, aligning rows based on left's column values and right's index labels
df_merged = pd.merge(left=df_econ, right=df_climate, how='inner', left_on='uc_id', right_index=True)
df_merged
# inner join: only retain rows where a match is found in both dataframes
# left join: retain all rows in left and bring in attributes from rows that matched in right
# right join: retain all rows in right and bring in attributes from rows that matched in left
# outer join: retain all rows in both dataframes
Explanation: 5. Merge and concatenate
5a. Merging DataFrames
Merging joins data together, aligned by the index. Assume here that we had two separate data sets, with unique uc_id identifiers.
End of explanation
# create two subset dataframes
df_us = df[df['country'] == 'united_states']
df_uk = df[df['country'] == 'united_kingdom']
# concatenate them together
df_us_uk = pd.concat([df_us, df_uk], sort=False)
df_us_uk
Explanation: 5b. Concatenating DataFrames
Concatenating smushes data together along some axis
End of explanation
# calculate per capita GDP then group the rows by region
df['gdp_percap'] = df['gdp_ppp'] / df['resident_pop']
groups = df.groupby('world_subregion')
# what is the median per capita GDP across the urban areas in each region?
groups['gdp_percap'].median().sort_values(ascending=False)
# look at several columns' medians by region
groups[['gdp_percap', 'avg_temperature', 'pop_density']].median()
# now it's your turn
# re-group the urban areas by country and find the highest then lowest urban area avg elevation in each country
Explanation: 6. Grouping and summarizing
End of explanation
# calculate resident population z-scores, vectorized
z = (df['resident_pop'] - df['resident_pop'].mean()) / df['resident_pop'].std()
z
Explanation: 7. Vectorization, Map, and Apply
Avoid slow iteration in pandas! Use vectorization (best) or map/apply (ok) when possible. These methods apply a function across elements, rows, and columns of a pandas DataFrame or Series. But they have some important and often confusing differences. Some tips:
avoid .iterrows() always
use vectorization wherever possible
.map() applies a function element-wise on a Series
.apply() works on a row or column basis on a DataFrame (specify the axis!), or element-wise on a Series
.applymap() works element-wise on an entire DataFrame
Let's see what that means in practice with some examples.
End of explanation
mean = df['resident_pop'].mean()
std = df['resident_pop'].std()
def calculate_zscore(x, mean=mean, std=std):
    return (x - mean) / std
    
# map the function to the series
z = df['resident_pop'].map(calculate_zscore)
Explanation: .map() applies a function element-wise on a series
End of explanation
mean = df['resident_pop'].mean()
std = df['resident_pop'].std()
# map a lambda function to the series
z = df['resident_pop'].map(lambda x: (x - mean) / std)
%%timeit
# check timings of vectorized vs map
z = (df['resident_pop'] - df['resident_pop'].mean()) / df['resident_pop'].std()
%%timeit 
mean = df['resident_pop'].mean()
std = df['resident_pop'].std()
z = df['resident_pop'].map(lambda x: (x - mean) / std)
Explanation: A lambda function is a simple, one-off, anonymous function. You can't call it again later because it doesn't have a name. It just lets you repeatedly perform some operation across a series of values (in our case, a column in our dataframe) using a minimal amount of code.
End of explanation
# find the difference between the min and max values in each column (ie, row-wise)
df_subset = df[['area', 'built_up_area', 'avg_elevation']]
df_subset.apply(lambda col: col.max() - col.min(), axis='rows')
# find the difference between the min and max values in each row (ie, column-wise)
df_subset.apply(lambda row: row.max() - row.min(), axis='columns')
# now it's your turn
# how would you vectorize the above .apply() methods to make the calculation faster?
# then compare the timings of your vectorized version to the .apply version above
Explanation: .apply() is like .map(), but it works on a row or column basis on an entire DataFrame (specify the axis)
End of explanation
# this uses applymap, but you could (should) vectorize it
# we'll see that next week
df_subset = df[['country', 'world_region', 'world_subregion']]
df_subset.applymap(lambda x: x.upper().replace('_', ' '))
Explanation: .applymap() works element-wise on an entire DataFrame. This is like doing a .map() to each column in the DataFrame.
End of explanation
df = df.reset_index().set_index(['country', 'core_city']).sort_index()
df
# index isn't unique, but doesn't need to be
# you could make it unique by adding uc_id as a 3rd level, for instance
df.index.is_unique
Explanation: 8. Hierarchical Indexing
A pandas index can comprise multiple values. It's very powerful, but gets tricky sometimes. See the docs.
In general, it's a good idea to sort your index for better querying performance later.
End of explanation
# select all urban areas in china (ie, first index level)
df.loc['china']
# or select rows by multiple index levels
# lots of unnamed core cities in china in this dataset
df.loc[('china', 'unnamed')]
# select every row with an unnamed core city in the dataset
# that is, only select by the 2nd level of the index
# the first : slices everything in the first index level, and the trailing : slices all columns
df.loc[pd.IndexSlice[:, ['unnamed']], :]
# select every row in argentina or spain with core city of cordoba
df.loc[(['argentina', 'spain'], ['cordoba']), :]
Explanation: So what's the benefit of multi-indexing like this? Your dataframe becomes a very fast look-up table. A good index is one that gives you a useful handle to select rows.
End of explanation
# now it's your turn
# multi-index the dataframe by world region, subregion, and uc_id
# use .loc to select all the rows from southern africa and eastern europe
Explanation: Stacking, unstacking, and pivoting are common ways to reshape data with a MultiIndex. Depending on the form your original data come in, they may be useful in the data prep phase to make your data easier to analyze. We'll discuss data prep, cleaning, and (review) visualization next week.
End of explanation
<END_TASK>