<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 Lesson 2
Step1:  Importing it takes some time
Has many functions like read_csv and uniq that help a lot
Step2:  05 - NumPy Arrays
Both Pandas and NumPy have special data structures for 1 D data
Numpy array is similar to Python list
Similarities
Access element by index
Access a range of elements
Use loops
Differences
Each element should have same type
Can have different types but it was designed for single data type
Convenient functions like mean and std
Step3:  Let's look at the element type of few array which numpy calls dtype
Step4:  |S11 means String with maximum length 11.
Step5:  07 - Vectorized Operations
Numpy supports Vectorized operations
A vector is a list of numbers
    
Addition of 2 vectors can be done in several ways. Different languages implement it differently
    
In case of NumPy it is an element wise addition
Step6:  09 - Multiplying by a Scalar
Multiplying by Scalar is scalar multiplied with each element of the array
Step7:  11 - Calculate Overall Completion Rate
More vectorized operations
Step8:  See this article for more information about bitwise operations.
In NumPy, a &amp; b performs a bitwise and of a and b. This is not necessarily the same as a logical and, if you wanted to see if matching terms in two integer vectors were non-zero. However, if a and b are both arrays of booleans, rather than integers, bitwise and and logical and are the same thing. If you want to perform a logical and on integer vectors, then you can use the NumPy function np.logical_and(a, b) or convert them into boolean vectors first.
Similarly, a | b performs a bitwise or, and ~a performs a bitwise not. However, if your arrays contain booleans, these will be the same as performing logical or and logical not. NumPy also has similar functions for performing these logical operations on integer-valued arrays.
In the solution, we may want to / (2.) instead of just / (2) . This is because in Python 2, dividing an integer by another integer (2) drops fractions, so if our inputs are also integers, we may end up losing information. If we divide by a float (2.) then we will definitely retain decimal values.
Step9:  13 - Standardizing Data
How does one data point compare to other data point?
One way to do this is to convert the data point to number of standard deviations from the mean
Step10:  15 - NumPy Index Arrays
Step11:  17 - + vs +=
Step12:  
Step13:  19 - In-Place vs Not In-Place
+= operates in-place while + does not
Step14:  slice refers to view of original array
21 - Pandas Series
Step15:  23 - Series Indexes
Step16:  Some people call countries[0] as indexing into array. But the instructor uses position 0 to avoid confusion. This is because in Pandas index and postion are not the same thing
Step17:  NumPy arrays are souped-up version of Python lists
Pandas Series is like a cross between a list and a dictionary
Step18:  25 - Vectorized Operations and Series Indexes
In NumPy arrays addition happens as per position
What happens if we add two Pandas series?
Step19:  Matching indexes were used to add the 2 series
Step20:  28 - Filling Missing Values - Solution
Step21:  29 - Pandas Series apply
So far we have used built-in functions like mean() and vectorized operations like +
apply takes a Series and a function and returns a new series applying the function on each element of the Series
Step22:  31 - Plotting in Pandas - Solution
<ASSISTANT_TASK:>
Python Code:
import pandas as pd
Explanation: Lesson 2: NumPy and Pandas for 1D Data
01 - Introduction
Will get familiar with 2 libraries - numpy and pandas
Writing Data Analysis code will be much easier.
Code runs faster
Analyse one dimensional data
02 - Gapminder Data
The data in this lesson was obtained from the site gapminder.org. The variables included are:
- Aged 15+ Employment Rate (%)
- Life Expectancy (years)
- GDP/capita (US$, inflation adjusted)
- Primary school completion (% of boys)
- Primary school completion (% of girls)
04 - One-Dimensional Data in NumPy and Pandas
End of explanation
import numpy as np
Explanation: Importing it takes some time
Has many functions like read_csv and uniq that help a lot
End of explanation
employments = pd.read_csv('employment_above_15.csv')
employments[0:5]
#Selecting a column and displaying its first 5 elements
employments.get('1991')[0:5]
employments.get('Country')[0:5]
def max_employment(countries, employment):    
    i = employment.argmax()
    return (countries[i], employment[i])
max_employment(employments.get('Country'), employments.get('2007'))
Explanation: 05 - NumPy Arrays
Both Pandas and NumPy have special data structures for 1 D data
Numpy array is similar to Python list
Similarities
Access element by index
Access a range of elements
Use loops
Differences
Each element should have same type
Can have different types but it was designed for single data type
Convenient functions like mean and std
End of explanation
countries = np.array(['Afghanistan','Albania','Algeria','Angola','Argentina','Armenia'])
employment = np.array([56.700001, 52.700001, 39.400002, 75.800003, 53.599998])
print countries.dtype
print employment.dtype
print np.array([0, 1, 2, 3]).dtype
print np.array([True, False, True]).dtype
print np.array(['AL', 'AK']).dtype
Explanation: Let's look at the element type of few array which numpy calls dtype
End of explanation
print employment.mean()
print employment.std()
print employment.max()
print employment.sum()
Explanation: |S11 means String with maximum length 11.
End of explanation
np.array([1, 2, 3]) + np.array([4, 5, 6])
Explanation: 07 - Vectorized Operations
Numpy supports Vectorized operations
A vector is a list of numbers
    
Addition of 2 vectors can be done in several ways. Different languages implement it differently
    
In case of NumPy it is an element wise addition
End of explanation
np.array([1, 2, 3]) * 3
Explanation: 09 - Multiplying by a Scalar
Multiplying by Scalar is scalar multiplied with each element of the array
End of explanation
np.array([1, 2, 3]) + np.array([4, 5, 6])
np.array([1, 2, 3]) + 1
np.array([1, 2, 3]) - np.array([7, 10, 15])
np.array([1, 2, 3]) - 1
np.array([1, 2, 3]) * np.array([4, 5, 6])
np.array([1, 2, 3]) * np.array([2])
#Throws error
#np.array([1, 2, 3]) * np.array([2, 3])
np.array([2, 3]) ** np.array([2, 3])
np.array([5, 6]) ** 2
Explanation: 11 - Calculate Overall Completion Rate
More vectorized operations
End of explanation
female_completion = pd.read_csv('female_completion_rate.csv')
male_completion = pd.read_csv('male_completion_rate.csv')
female_completion[0:5]
male_completion[0:5]
female = np.array([56.0, 23.0, 65.0])
male = np.array([23.0, 45.0, 22.0])
def overall_completion_rate(female_completion, male_completion):
    return (female_completion + male_completion) / 2
overall_completion_rate(female, male)
Explanation: See this article for more information about bitwise operations.
In NumPy, a &amp; b performs a bitwise and of a and b. This is not necessarily the same as a logical and, if you wanted to see if matching terms in two integer vectors were non-zero. However, if a and b are both arrays of booleans, rather than integers, bitwise and and logical and are the same thing. If you want to perform a logical and on integer vectors, then you can use the NumPy function np.logical_and(a, b) or convert them into boolean vectors first.
Similarly, a | b performs a bitwise or, and ~a performs a bitwise not. However, if your arrays contain booleans, these will be the same as performing logical or and logical not. NumPy also has similar functions for performing these logical operations on integer-valued arrays.
In the solution, we may want to / (2.) instead of just / (2) . This is because in Python 2, dividing an integer by another integer (2) drops fractions, so if our inputs are also integers, we may end up losing information. If we divide by a float (2.) then we will definitely retain decimal values.
End of explanation
def standardize_data(values):
    return (values - values.mean()) / values.std()
Explanation: 13 - Standardizing Data
How does one data point compare to other data point?
One way to do this is to convert the data point to number of standard deviations from the mean
End of explanation
def mean_time_for_paid_students(time_spent, days_to_cancel):
    return time_spent[days_to_cancel >= 7].mean()
Explanation: 15 - NumPy Index Arrays
End of explanation
a = np.array([1, 2, 3, 4])
b = a
a += np.array([1, 1, 1, 1]) #Difference here
print b
Explanation: 17 - + vs +=
End of explanation
a = np.array([1, 2, 3, 4])
b = a
a = a + np.array([1, 1, 1, 1]) #Difference here
print b
Explanation: 
End of explanation
a = np.array([1, 2, 3, 4, 5])
slice = a[:3]
slice[0] = 100
a
Explanation: 19 - In-Place vs Not In-Place
+= operates in-place while + does not
End of explanation
def variable_correlation(variable1, variable2):
    both_above = (variable1 > variable1.mean()) & \
                 (variable2 > variable2.mean())
    both_below = (variable1 < variable1.mean()) & \
                 (variable2 < variable2.mean())
    
    is_same_direction = both_above | both_below
    num_same_direction = is_same_direction.sum()
    
    num_different_direction = len(variable1) - num_same_direction
    
    return (num_same_direction, num_different_direction)
Explanation: slice refers to view of original array
21 - Pandas Series
End of explanation
s = pd.Series([1, 2, 3, 4])
s.describe()
countries = np.array(['Albania', 'Algeria', 'Andorra', 'Angola'])
life_expectancy = np.array([74.7, 75., 83.4, 57.6])
life_expectancy
Explanation: 23 - Series Indexes
End of explanation
life_expectancy = pd.Series([74.7, 75., 83.4, 57.6],
                           index = ['Albania', 
                                    'Algeria', 
                                    'Andorra', 
                                    'Angola'])
life_expectancy
Explanation: Some people call countries[0] as indexing into array. But the instructor uses position 0 to avoid confusion. This is because in Pandas index and postion are not the same thing
End of explanation
#Access by index
life_expectancy.loc['Angola']
#If we don't specify index then automatically adds index 0, 1, 2, ...
pd.Series([74.7, 75., 83.4, 57.6])
#Access element by position
print life_expectancy.iloc[0]
#same as
print life_expectancy[0]
def max_employment(employment):
    max_country = employment.argmax()
    max_value = employment.loc[max_country]
    
    return (max_country, max_value)
Explanation: NumPy arrays are souped-up version of Python lists
Pandas Series is like a cross between a list and a dictionary
End of explanation
s1 = pd.Series([1, 2, 3, 4], index = ['a', 'b', 'c', 'd'])
s2 = pd.Series([10, 20, 30, 40], index = ['a', 'b', 'c', 'd'])
s1
s2
s1 + s2
# Index are in different order
s3 = pd.Series([10, 20, 30, 40], index = ['b', 'd', 'a', 'c'])
s3
s1 + s3
Explanation: 25 - Vectorized Operations and Series Indexes
In NumPy arrays addition happens as per position
What happens if we add two Pandas series?
End of explanation
s4 = pd.Series([10, 20, 30, 40], index = ['c', 'd', 'e', 'f'])
s4
s1 + s4
#If we don't want to show NaN in our solution
(s1 + s4).dropna()
Explanation: Matching indexes were used to add the 2 series
End of explanation
#If we want to give a default value
s1.add(s4, fill_value=0)
Explanation: 28 - Filling Missing Values - Solution
End of explanation
names = pd.Series([
        'Andre Agassi',
        'Barry Bonds',
        'Christopher Columbus',
        'Daniel Defoe'
    ])
def reverse_name(name):
    split_name = name.split(" ")
    return "{}, {}".format(split_name[1], split_name[0])
reverse_name(names.iloc[0])
def reverse_names(names):
    return names.apply(reverse_name)
reverse_names(names)
Explanation: 29 - Pandas Series apply
So far we have used built-in functions like mean() and vectorized operations like +
apply takes a Series and a function and returns a new series applying the function on each element of the Series
End of explanation
employment = pd.read_csv('employment_above_15.csv', index_col = 'Country')
female_completion = pd.read_csv('female_completion_rate.csv', index_col = 'Country')
male_completion = pd.read_csv('male_completion_rate.csv', index_col = 'Country')
life_expectancy = pd.read_csv('life_expectancy.csv', index_col = 'Country')
gdp_per_capita = pd.read_csv('gdp_per_capita.csv', index_col = 'Country')
_country = 'United States'
employment_country = employment.loc[_country]
female_completion_country = female_completion.loc[_country]
male_completion_country = male_completion.loc[_country]
life_expectancy_country = life_expectancy.loc[_country]
gdp_per_capita_country = gdp_per_capita.loc[_country]
%pylab inline
employment_country.plot()
female_completion_country.plot()
male_completion_country.plot()
life_expectancy_country.plot()
gdp_per_capita_country.plot()
Explanation: 31 - Plotting in Pandas - Solution
End of explanation
<END_TASK>