<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 Toxic Armories
The Toxic Armories investigation produced a comprehensive dataset over 1000 armories across the US, as well as thousands of pages of documents requested through Freedom of Information Act requests.
The Oregonian has made both the inspection reports and the dataset publicly available. The purpose of this notebook is to demonstrate ways you can make use of both.
Step1:  Inspection Reports and DocumentCloud
All of the inspection reports we have as of publication and all that we receive later will be free to the public through the excellent open source journalism resource DocumentCloud.
The ID corresponding to the armories project is 30072. You can either use the DocumentCloud API or use the web interface to find more, but the former allows you to access many documents at once.
Step2:  Here I convert the relatively human-readable dict to the right format for querying DocumentCloud
Step3:  The Data
Over 18 months, reporter Rob Davis took notes on all of the armories he was investigating, and also tracked them in a spreadsheet. Through collaboration with other data team members, these notes were structured into a compromise between what is human-readable and what is structured for visualization. The dataset that informed his article and underlies the interactive at publication time follows.
Step4:  Each row in df_data corresponds to one armory. Below I have selected the row with data on a Portland, OR armory about two miles from the Portland International Airport.
What these variables mean
Step5:  Below I change "Unknown" to "NaN" so that, when I try to calculate counts for different variables, it is evident that this information is missing
Step6:  If you aren't already familiar with programming
Step7:  Python's groupby behaves much like Excel's pivot tables, and is also a process in SQL.
Step8:  Below I've made a new dataframe to examine what proportion of armories in a given state had a firing range, and what proportion had lead.
Step9:  I counted the number of unique IDs that The Oregonian had (Oregonian ID) for armories within each state, so I'll rename that column to Armory Count for the sake of clarity.
Step10:  Here I recombine the two dataframes because one had to be a calculation that counted the number of armories, while df_state_values allowed me to sum the 1s that I had generated for Yes for a given variable, e.g. Inspection conducted?
Step11:  I'm going to repurpose df_state_count to more easily show the results of some calculations
Step12:  States ordered by what proportion of armories were inspected
This is an unwieldy table by now, but for the sake of comprehensiveness I'll display it in full.
Step13:  It's verbose and it isn't pretty, but plotting is also built into pandas!
<ASSISTANT_TASK:>
Python Code:
import json
import urllib.parse
import numpy as np
import pandas as pd
import requests
%matplotlib inline
Explanation: Toxic Armories
The Toxic Armories investigation produced a comprehensive dataset over 1000 armories across the US, as well as thousands of pages of documents requested through Freedom of Information Act requests.
The Oregonian has made both the inspection reports and the dataset publicly available. The purpose of this notebook is to demonstrate ways you can make use of both.
End of explanation
# It might be overkill, but I figured it best
# for legibility to separate query arguments as a dict
params_dict = {
    "q":"projectid:30072",
    "per_page":"1000"
}
Explanation: Inspection Reports and DocumentCloud
All of the inspection reports we have as of publication and all that we receive later will be free to the public through the excellent open source journalism resource DocumentCloud.
The ID corresponding to the armories project is 30072. You can either use the DocumentCloud API or use the web interface to find more, but the former allows you to access many documents at once.
End of explanation
params_encoded = urllib.parse.urlencode(params_dict)
r = requests.get('https://www.documentcloud.org/api/search.json', params=params_encoded)
r.json()['documents'][0]
documentcloud_df = pd.read_json(json.dumps(r.json()['documents']))
documentcloud_df.head()
Explanation: Here I convert the relatively human-readable dict to the right format for querying DocumentCloud:
End of explanation
df = pd.read_csv('armories_data - 20161201.csv',dtype={'Oregonian ID':'str','Inspection year':'str'})
Explanation: The Data
Over 18 months, reporter Rob Davis took notes on all of the armories he was investigating, and also tracked them in a spreadsheet. Through collaboration with other data team members, these notes were structured into a compromise between what is human-readable and what is structured for visualization. The dataset that informed his article and underlies the interactive at publication time follows.
End of explanation
df.ix[975]
Explanation: Each row in df_data corresponds to one armory. Below I have selected the row with data on a Portland, OR armory about two miles from the Portland International Airport.
What these variables mean
End of explanation
df.head()
Explanation: Below I change "Unknown" to "NaN" so that, when I try to calculate counts for different variables, it is evident that this information is missing:
Note: NaN (not a number) is the standard missing data marker used in pandas
Source: http://pandas-docs.github.io/pandas-docs-travis/dsintro.html#series
End of explanation
df_modified = df.replace(to_replace=['Unknown','Yes','No'],value=[np.nan,1,0])
df_modified.head()
Explanation: If you aren't already familiar with programming: below I change the character values from "Yes" and "No," called Strings in Python and many other programming languages, to numbers (1,0) so that I can calculate rates of inspection or rates of lead presence later.
End of explanation
df_states = df_modified[['State','Oregonian ID','Lead present?','Had firing range?']].groupby('State').sum().reset_index()
Explanation: Python's groupby behaves much like Excel's pivot tables, and is also a process in SQL.
End of explanation
df_state_count = df_modified[['State','Oregonian ID']].groupby('State').count().reset_index()
Explanation: Below I've made a new dataframe to examine what proportion of armories in a given state had a firing range, and what proportion had lead.
End of explanation
df_state_count.rename(columns={'Oregonian ID':'Armory Count'},inplace=True)
df_state_values = df_modified[['State','Inspection conducted?','Inspection report available?','Lead present?','Lead present outside range?','Had firing range?']].groupby('State').sum().reset_index()
Explanation: I counted the number of unique IDs that The Oregonian had (Oregonian ID) for armories within each state, so I'll rename that column to Armory Count for the sake of clarity.
End of explanation
df_states = pd.merge(df_state_count,df_state_values)
Explanation: Here I recombine the two dataframes because one had to be a calculation that counted the number of armories, while df_state_values allowed me to sum the 1s that I had generated for Yes for a given variable, e.g. Inspection conducted?
End of explanation
df_state_values['Rate of lead in state'] = round(df_states['Lead present?']/df_states['Armory Count'],2)
df_state_values['Rate of inspection'] = round(df_states['Inspection conducted?']/df_states['Armory Count'],2)
Explanation: I'm going to repurpose df_state_count to more easily show the results of some calculations:
End of explanation
df_state_values.sort_values(by=['Rate of inspection'],ascending=False)
Explanation: States ordered by what proportion of armories were inspected
This is an unwieldy table by now, but for the sake of comprehensiveness I'll display it in full.
End of explanation
df_state_values[['Rate of inspection']].sort_values(
    by='Rate of inspection', ascending=False).plot(
    kind='bar',
    title='Rate of armory inspection by state',
    legend=False,
    x=df_state_values['State'])
Explanation: It's verbose and it isn't pretty, but plotting is also built into pandas!
End of explanation
<END_TASK>