<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 Manipulating Data in Python
Laila A. Wahedi
Massive Data Institute Postdoctoral Fellow <br>McCourt School of Public Policy
Follow along
Step1:  Numbers
Step2:  Lists
Declare with
Step3:  Zero indexed
Step4:  Lists are Mutable
Step5:  Dictionaries
key value pairs indexed by key
Declared by {key
Step6:  Add key value pairs later using indexing with brackets []
You can store data type you want
Step7:  Sets
Like a list, but
Step8:  Tuples
Like an ordered list, but can't be changed
Declare with ()
Step9:  Our First Algorithm
Step10:  Our First Algorithm
Step11:  Adding Choices
<img src="if_algorithm.png">
Writing Instructions Python Can Understand
Step12:  Compare values with
Step13:  Booleans and Indicators
booleans are indicators of True or False
In Python3, True = 1, and False = 0
Step14:  None is a null value
None evaluates to false in an if statement. 
If statements
Step15:  Play around yourself
Step16:  Repetition
<img src="loops.png">
While loops
Do something in block of code until condition met
Make sure you change the condition in every loop, or it will go forever
Step17:  For loops
repeat block of code for
Step18:  Put them together
Step19:  Learn More
Step20:  Pandas Data Frames
Using Documentation
Pandas website
Stack Overflow
Copy errors into google
Look up syntax differences with R
Load Data from a comma separated file
Start by googling it
Step21:  Look at the data
Step22:  Rename things and adjust values
Step23:  Set a useful index
Step24:  Slicing
Get specific values from the dataframe. 
Pandas has several slice operators. 
iloc can be used to index the row by ordered integer. i.e. first row is 0, second row is 1, etc. Use this option sparingly. Better practice to use the index you have created. 
loc uses the named index and columns. 
Index using [row, columns]
For multiple columns, put your column names in a list
Use 
Step25:  Look at the data
Step26:  Slicing Using Conditionals
Put conditionals in parentheses
Stack multiple conditionals using
Step27:  Handling Missing Values
First lets make some
Step28:  Handling Missing Values
We could index by them
Step29:  Handling Missing Values
We could fill them
Step30:  Making New Columns
Assign values to a new column based on other columns
Step31:  Reindexing
Step32:  Set a multi-index
Step33:  Using the new index, make a new dataframe
Note the new slicing operator for multi-index
Step34:  Warning
Step35:  What happened?
copied_df changed when little_df changed. 
Let's fix that
Step36:  Saving
Unlike R or Stata, can't just save your workspace
Save as a csv now that we have the data we want
Pickle a variable to recreate it without having to reset indexes, etc.
Step37:  Next time
Step38:  Joins
Same as SQL, inner and outer
Step39:  Concatenate
Stack dataframes on top of one another
Stack dataframes beside one another
<ASSISTANT_TASK:>
Python Code:
my_string = 'Hello World'
print(my_string)
Explanation: Manipulating Data in Python
Laila A. Wahedi
Massive Data Institute Postdoctoral Fellow <br>McCourt School of Public Policy
Follow along: Wahedi.us, Current Presentation
Installing packages:
On a Mac:
Open terminal 
On Windows:
Type cmd into the start menu
What is this?
Like an explorer window. The text at the start of the line tells you where you are. 
To see what's there, type: 
On Mac: ls (That's an L, not an i)
On Windows: dir
Installing Packages:
Install Packages
Install some packages so they are done by the time we need them.
    Type:
pip install pandas
When that's done, type: 
pip install matplotlib
pip install pickle
pip install statsmodels
Note: if you installed the Continuum Python distribution, you may already have some of these packages installed. 
What just happened?
Pip is a download manager, it automatically looks online to find the package and installs it on your computer
Similar to the CRAN Mirror
Bonus: if you followed the link I sent to install Anaconda, you have the conda download manager too. 
conda install works similarly to pip, and also works on many R packages when used with Jupyter notebooks. 
Opening Jupyter
Open up another terminal/command prompt while your packages download
Navigate to the directory you want your script to be in
Type ls to see what folders are in the current folder
Type cd .. to go one folder up
Type cd folder_name to go to a sub-folder
Type mkdir folder_name to make a folder
Type: 
jupyter notebook
What happened?
A browser window popped up
Address something along the lines of localhost:8888/tree
Address is listed in the terminal/command prompt 
Go to this address from any browser to open your tree
Your Jupyter Tree
You should be in the folder you navigated to in terminal/cmd
You can make folders here too
Top right, select new notebook, python3
Open this notebook:
Move the file into the directory for your Jupyter Tree
Select the file from teh Jupyter Tree
A new tab should open in your browser.
Add new cells to take notes as you go along. 
Change the type of note cells to Markdown
Some Python Basics
What is code?
Set of clear instructions to the computer
Start with basic set of building blocks
Put those building blocks together 
Goal of programming:
Break your task into building blocks 
Put building blocks together into an <b> algorithm </b> 
Our tasks:
Move and manipulate data using these building blocks. 
Building Blocks
<img src="building_blocks.png">
* Ball and urn example
Our First Algorithm
<img src="algorithm1.png">
Basic Data Structures
Strings
Text variables
Declare with either '', or ""
End of explanation
my_int = 2
my_float = 2.2
new_float = my_int+my_float
print(new_float)
type(new_float)
Explanation: Numbers:
ints: integers
floats: numbers with decimals 
Combine them to get a new float
End of explanation
my_list = [0,1,2,3,4]
Explanation: Lists
Declare with: []
Ordered list of objects
End of explanation
print(my_list[1])
Explanation: Zero indexed
End of explanation
my_list[2]='hello'
print(my_list)
Explanation: Lists are Mutable:
End of explanation
my_dictionary = {'apple':4,
                'pear':'yum'}
print(my_dictionary['apple'])
Explanation: Dictionaries
key value pairs indexed by key
Declared by {key:value}
End of explanation
my_dictionary['numbers'] = my_list
print(my_dictionary['numbers'])
Explanation: Add key value pairs later using indexing with brackets []
You can store data type you want
End of explanation
my_set = {'thing1','thing2','cat in hat','thing1', 4,4}
print(my_set)
Explanation: Sets
Like a list, but:
Contains unique values
Unordered
Declare with {}
End of explanation
my_tuple = (1,3,2)
print(my_tuple)
Explanation: Tuples
Like an ordered list, but can't be changed
Declare with ()
End of explanation
# Declare Data
my_data = 'hello '
my_other_data = 'world'
#Manipulate it
manipulated_data = my_data+my_other_data
#Output it:
print(manipulated_data)
Explanation: Our First Algorithm: Strings
<img src="algorithm1.png">
End of explanation
# Declare Data
my_data =  1
my_other_data = 5
#Manipulate it
manipulated_data = 1/5
#Output it:
print(manipulated_data)
Explanation: Our First Algorithm: Numbers
<img src="algorithm1.png">
End of explanation
my_variable = 5
print(my_variable)
print(my_variable == 5)
Explanation: Adding Choices
<img src="if_algorithm.png">
Writing Instructions Python Can Understand:
Whitespace matters
tabs indicate a block of code within an if statement or loop
: marks the start of the block
\ for a linebreak mid-line
Line breaks allowed inside (), [], {}
If statements
= to assign
Like <- in R
== to evaluate
End of explanation
print(my_variable > 6)
print(my_variable in [1,4,7])
Explanation: Compare values with: 
<, <=, >, >=, ==, !=, in, not in,
End of explanation
True + True
Explanation: Booleans and Indicators
booleans are indicators of True or False
In Python3, True = 1, and False = 0
End of explanation
my_bool = 'ice cream'
if my_bool == 'ice cream': 
    print('yay')
elif my_bool == 'cake':
    print('woo!')
else:
    print('Woe and great tragedy!')
Explanation: None is a null value
None evaluates to false in an if statement. 
If statements:
Used to change behavior depending on conditionals
Declare the statement
Declare the conditional action within a code block, or indentation:
Declare alternative actions in else
Stack conditionals with elif
End of explanation
check = True
# check = False
# check = None
# check = 'monkey'
# check = 0
# check = 10
print('Check is:', check)
if check == 'monkey':
    print('banana')
elif check:
    print('yes')
else:
    print('no')
    
if 1 not in [1,2,3]:
    print('not not in')
if 1 in [1,2,3]:
    print('in')
Explanation: Play around yourself:
End of explanation
n = 0
while n < 5:
    print(n)
    n= n+1
Explanation: Repetition
<img src="loops.png">
While loops
Do something in block of code until condition met
Make sure you change the condition in every loop, or it will go forever
End of explanation
print('use a range:')
for i in range(3):
    print(i)
print('use a range slice:')
for i in range(3,6):
    print(i)
print('iterate throubh a list:')
for i in my_list:
    print(i)
Explanation: For loops
repeat block of code for:
a certain number of iterations
For every element in a list
range() gives you an iterator
End of explanation
my_list = [0,1,'cat',None,'frog',3]
animals = []
nums = []
for i in my_list:
    if type(i)==str:
        animals.append(i)
    elif type(i)==int:
        nums.append(i)
    else:
        pass
print(animals)
print(nums)
Explanation: Put them together:
Play around on your own
End of explanation
import pandas as pd
import matplotlib.pyplot as plt
%matplotlib inline
# from ggplot import *
import pickle
import statsmodels.api as sm
Explanation: Learn More:
These are the basic building blocks for scripting. To learn more about how to put them together to greater effect:
Take an introductory computer science course online
Check out GU Women Coders. 
Can't make it in person? Their website has lots of great resources.
http://guwecode.georgetown.domains/
Import packages
Like library(package) in R
Pandas is a dataframe data structure like dataframes in R
matplotlib is a plotting package similar to plotting in Matlab
you can view plots inline in pandas notebooks with the inline command
ggplot is a plotting package built on matplotlib like ggplot2 in R
Pickle lets you save your workspace
Statsmodels contains many of the statistical models you know and love
End of explanation
baad_covars = pd.read_csv('BAAD_1_Lethality_Data.tab',sep='\t')
Explanation: Pandas Data Frames
Using Documentation
Pandas website
Stack Overflow
Copy errors into google
Look up syntax differences with R
Load Data from a comma separated file
Start by googling it: http://lmgtfy.com/?q=pandas+load+csv
We will use the Big Allied and Dangerous Data from START
https://dataverse.harvard.edu/file.xhtml?fileId=2298519&version=RELEASED&version=.0
End of explanation
baad_covars.head()
Explanation: Look at the data
End of explanation
baad_covars.rename(columns = {'cowmastercountry':'country',
                              'masterccode':'ccode',
                       'mastertccode3606':'group_code',
                       'fatalities19982005':'fatalities'},
            inplace = True)
baad_covars.replace({'country':{'United States of America':'US'}},
            inplace = True)
print('Dimensions: ',baad_covars.shape)
baad_covars.head()
Explanation: Rename things and adjust values
End of explanation
#Set the index
baad_covars.set_index(['group_code'],inplace = True)
baad_covars.head()
Explanation: Set a useful index
End of explanation
baad_covars.loc[:, 'fatalities'].head()
Explanation: Slicing
Get specific values from the dataframe. 
Pandas has several slice operators. 
iloc can be used to index the row by ordered integer. i.e. first row is 0, second row is 1, etc. Use this option sparingly. Better practice to use the index you have created. 
loc uses the named index and columns. 
Index using [row, columns]
For multiple columns, put your column names in a list
Use : for all values
Notice that the output keeps the index names.
End of explanation
baad_covars.loc[:,['OrgAge']].plot.density()
print(baad_covars.loc[:,['OrgAge']].mean())
baad_covars.loc[:,['fatalities']].plot.hist(bins=20)
Explanation: Look at the data
End of explanation
baad_covars.loc[(baad_covars.fatalities>1) | (baad_covars.degree>=1),
               ['group','country']].head()
Explanation: Slicing Using Conditionals
Put conditionals in parentheses
Stack multiple conditionals using:
& when both conditions must always apply
| when at least one condition must apply
End of explanation
baad_covars.loc[(baad_covars.fatalities>1) | (baad_covars.degree>=1),
               ['terrStrong']] = None
baad_covars.loc[(baad_covars.fatalities>1) | (baad_covars.degree>=1),
               ['terrStrong']].head()
Explanation: Handling Missing Values
First lets make some:
End of explanation
baad_covars.loc[baad_covars.terrStrong.isnull(),'terrStrong'].head()
Explanation: Handling Missing Values
We could index by them
End of explanation
baad_covars['terrStrong'] = baad_covars.terrStrong.fillna(-77)
baad_covars.terrStrong.head()
Explanation: Handling Missing Values
We could fill them:
End of explanation
baad_covars['big'] = 0
baad_covars.loc[(baad_covars.fatalities>1) | 
                (baad_covars.degree>=1),
               'big']=1
baad_covars.big.head()
Explanation: Making New Columns
Assign values to a new column based on other columns:
End of explanation
baad_covars.reset_index(inplace=True)
baad_covars.head()
Explanation: Reindexing: Pop the index out without losing it
End of explanation
baad_covars.set_index(['group','country'],inplace = True)
baad_covars.head()
Explanation: Set a multi-index
End of explanation
indonesia_grps = baad_covars.xs('Indonesia',level = 'country',drop_level=False)
indonesia_grps = indonesia_grps.loc[indonesia_grps.fatalities>=1,['degree','ContainRelig',
                                            'ContainEthno','terrStrong',
                                            'ordsize','OrgAge']]
indonesia_grps.head()
Explanation: Using the new index, make a new dataframe
Note the new slicing operator for multi-index
End of explanation
little_df = pd.DataFrame([1,2,3,4,5],columns = ['A'])
little_df['B']=[0,1,0,1,1]
copied_df = little_df
print('before:')
print(copied_df)
little_df.loc[little_df.A == 3,'B'] = 7
print('after')
copied_df
Explanation: Warning: Making copies
If you set a variable as equal to an object, Python creates a reference rather than copying the whole object. More efficient, unless you really want to make a copy
End of explanation
import copy
little_df = pd.DataFrame([1,2,3,4,5],columns = ['A'])
little_df['B']=[0,1,0,1,1]
copied_df = little_df.copy()
print('before:')
print(copied_df)
little_df.loc[little_df.A == 3,'B'] = 7
print('after')
copied_df
Explanation: What happened?
copied_df changed when little_df changed. 
Let's fix that: import "copy"
End of explanation
indonesia_grps.to_csv('indonesia.csv')
pickle.dump(indonesia_grps, open('indonesia.p','wb'))
indonesia_grps = pickle.load(open('indonesia.p','rb'))
Explanation: Saving
Unlike R or Stata, can't just save your workspace
Save as a csv now that we have the data we want
Pickle a variable to recreate it without having to reset indexes, etc.
End of explanation
C = pd.DataFrame(['apple','orange','grape','pear','banana'],
                 columns = ['C'],
                 index = [2,4,3,0,1])
little_df['C'] = C
little_df
Explanation: Next time:
Scrape Data from the internet
Clean the data
Merge that data into our data
Run a basic stats and ML model
Until then, here's some reference code on merging your data sets
Merging and Concatenating
Merges automatically if shared index
End of explanation
C = pd.DataFrame(['apple','orange','grape','apple'],
                 columns = ['C'],
                 index = [2,4,3,'a'])
C['cuts']=['slices','wedges','whole','spirals']
print('C:')
print(C)
print('Inner: Intersection')
print(little_df.merge(right=C,
                how='inner',
                on=None,
                left_index = True,
                right_index =True))
print('Outer: Keep all rows')
print(little_df.merge(right=C,
                how='outer',
                on=None,
                left_index = True,
                right_index =True))
print('Left: Keep little_df')
print(little_df.merge(right=C,
                how='left',
                on=None,
                left_index = True,
                right_index =True))
print('Right: Keep C')
print(little_df.merge(right=C,
                how='right',
                on=None,
                left_index = True,
                right_index =True))
print('Outer, merging on column instead of index')
print(little_df.merge(right=C,
                how='outer',
                on='C',
                left_index = True,
                right_index =True))
Explanation: Joins
Same as SQL, inner and outer
End of explanation
add_df = pd.DataFrame({'A':[6],'B':[7],'C':'peach'},index= ['p'])
little_df = pd.concat([little_df,add_df])
little_df
Explanation: Concatenate
Stack dataframes on top of one another
Stack dataframes beside one another
End of explanation
<END_TASK>