<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 Accessing Databases via Web APIs
Step1:  1. Constructing API GET Request
In the first place, we know that every call will require us to provide
Step2:  You often want to send some sort of data in the URL’s query string. This data tells the API what information you want. In our case, we want articles about Duke Ellington. Requests allows you to provide these arguments as a dictionary, using the params keyword argument. In addition to the search term q, we have to put in the api-key term.
Step3:  Now we're ready to make the request. We use the .get method from the requests library to make an HTTP GET Request.
Step4:  Now, we have a response object called r. We can get all the information we need from this object. For instance, we can see that the URL has been correctly encoded by printing the URL. Click on the link to see what happens.
Step5:  Click on that link to see it returns!
Challenge 1
Step6:  Challenge 2
Step7:  2. Parsing the response text
We can read the content of the server’s response using .text
Step8:  What you see here is JSON text, encoded as unicode text. JSON stands for "Javascript object notation." It has a very similar structure to a python dictionary -- both are built on key/value pairs. This makes it easy to convert JSON response to a python dictionary.
Step9:  That looks intimidating! But it's really just a big dictionary. Let's see what keys we got in there.
Step10:  That looks what we want! Let's put that in it's own variable.
Step11:  3. Putting everything together to get all the articles.
That's great. But we only have 10 items. The original response said we had 93 hits! Which means we have to make 93 /10, or 10 requests to get them all. Sounds like a job for a loop! 
But first, let's review what we've done so far.
Step12:  Challenge 3
Step13:  4. Formatting
Let's take another look at one of these documents.
Step14:  This is all great, but it's pretty messy. What we’d really like to to have, eventually, is a CSV, with each row representing an article, and each column representing something about that article (header, date, etc). As we saw before, the best way to do this is to make a lsit of dictionaries, with each dictionary representing an article and each dictionary representing a field of metadata from that article (e.g. headline, date, etc.) We can do this with a custom function
Step15:  Challenge 4 Collect more fields
Edit the function above so that we include the lead_paragraph and word_count fields.
HINT
Step16:  5. Exporting
We can now export the data to a CSV.
Step17:  Capstone Challenge
Using what you learned, tell me if Chris' claim (i.e. that Duke Ellington has gotten more popular lately) holds water.
<ASSISTANT_TASK:>
Python Code:
# Import required libraries
import requests
import json
from __future__ import division
import math
import csv
import matplotlib.pyplot as plt
Explanation: Accessing Databases via Web APIs
End of explanation
# set key
key="be8992a420bfd16cf65e8757f77a5403:8:44644296"
# set base url
base_url="http://api.nytimes.com/svc/search/v2/articlesearch"
# set response format
response_format=".json"
Explanation: 1. Constructing API GET Request
In the first place, we know that every call will require us to provide:
a base URL for the API, 
some authorization code or key, and 
a format for the response. 
So let's put store those in some variables.
Use the following demonstration keys for now, but in the future, get your own!
ef9055ba947dd842effe0ecf5e338af9:15:72340235
25e91a4f7ee4a54813dca78f474e45a0:15:73273810
e15cea455f73cc47d6d971667e09c31c:19:44644296
b931c838cdb745bbab0f213cfc16b7a5:12:44644296
1dc1475b6e7d5ff5a982804cc565cd0b:6:44644296
18046cd15e21e1b9996ddfb6dafbb578:4:44644296
be8992a420bfd16cf65e8757f77a5403:8:44644296
End of explanation
# set search parameters
search_params = {"q":"Duke Ellington",
                 "api-key":key}       
Explanation: You often want to send some sort of data in the URL’s query string. This data tells the API what information you want. In our case, we want articles about Duke Ellington. Requests allows you to provide these arguments as a dictionary, using the params keyword argument. In addition to the search term q, we have to put in the api-key term.
End of explanation
# make request
r = requests.get(base_url+response_format, params=search_params)
Explanation: Now we're ready to make the request. We use the .get method from the requests library to make an HTTP GET Request.
End of explanation
print(r.url)
Explanation: Now, we have a response object called r. We can get all the information we need from this object. For instance, we can see that the URL has been correctly encoded by printing the URL. Click on the link to see what happens.
End of explanation
# set date parameters here
# Uncomment to test
# r = requests.get(base_url+response_format, params=search_params)
# print(r.url)
Explanation: Click on that link to see it returns!
Challenge 1:  Adding a date range
What if we only want to search within a particular date range? The NYT Article Api allows us to specify start and end dates.
Alter the search_params code above so that the request only searches for articles in the year 2015.
You're gonna need to look at the documentation here to see how to do this.
End of explanation
# set page parameters here
# Uncomment to test
# r = requests.get(base_url+response_format, params=search_params)
# print(r.url)
Explanation: Challenge 2:  Specifying a results page
The above will return the first 10 results. To get the next ten, you need to add a "page" parameter. Change the search parameters above to get the second 10 resuls.
End of explanation
# Inspect the content of the response, parsing the result as text
response_text= r.text
print(response_text[:1000])
Explanation: 2. Parsing the response text
We can read the content of the server’s response using .text
End of explanation
# Convert JSON response to a dictionary
data = json.loads(response_text)
# data
Explanation: What you see here is JSON text, encoded as unicode text. JSON stands for "Javascript object notation." It has a very similar structure to a python dictionary -- both are built on key/value pairs. This makes it easy to convert JSON response to a python dictionary.
End of explanation
print(data.keys())
# this is boring
data['status']
# so is this
data['copyright']
# this is what we want!
# data['response']
data['response'].keys()
data['response']['meta']['hits']
# data['response']['docs']
type(data['response']['docs'])
Explanation: That looks intimidating! But it's really just a big dictionary. Let's see what keys we got in there.
End of explanation
docs = data['response']['docs']
docs[0]
Explanation: That looks what we want! Let's put that in it's own variable.
End of explanation
# set key
key="ef9055ba947dd842effe0ecf5e338af9:15:72340235"
# set base url
base_url="http://api.nytimes.com/svc/search/v2/articlesearch"
# set response format
response_format=".json"
# set search parameters
search_params = {"q":"Duke Ellington",
                 "api-key":key,
                 "begin_date":"20150101", # date must be in YYYYMMDD format
                 "end_date":"20151231"}
# make request
r = requests.get(base_url+response_format, params=search_params)
    
# convert to a dictionary
data=json.loads(r.text)
    
# get number of hits
hits = data['response']['meta']['hits']
print("number of hits: ", str(hits))
    
# get number of pages
pages = int(math.ceil(hits/10))
    
# make an empty list where we'll hold all of our docs for every page
all_docs = [] 
    
# now we're ready to loop through the pages
for i in range(pages):
    print("collecting page", str(i))
        
    # set the page parameter
    search_params['page'] = i
        
    # make request
    r = requests.get(base_url+response_format, params=search_params)
    
    # get text and convert to a dictionary
    data=json.loads(r.text)
        
    # get just the docs
    docs = data['response']['docs']
        
    # add those docs to the big list
    all_docs = all_docs + docs
len(all_docs)
Explanation: 3. Putting everything together to get all the articles.
That's great. But we only have 10 items. The original response said we had 93 hits! Which means we have to make 93 /10, or 10 requests to get them all. Sounds like a job for a loop! 
But first, let's review what we've done so far.
End of explanation
# DEFINE YOUR FUNCTION HERE
# uncomment to test
# get_api_data("Duke Ellington", 2014)
Explanation: Challenge 3: Make a function
Turn the code above into a function that inputs a search term and a year, and returns all the documents containing that search term in that year.
End of explanation
all_docs[0]
Explanation: 4. Formatting
Let's take another look at one of these documents.
End of explanation
def format_articles(unformatted_docs):
    '''
    This function takes in a list of documents returned by the NYT api 
    and parses the documents into a list of dictionaries, 
    with 'id', 'header', and 'date' keys
    '''
    formatted = []
    for i in unformatted_docs:
        dic = {}
        dic['id'] = i['_id']
        dic['headline'] = i['headline']['main']
        dic['date'] = i['pub_date'][0:10] # cutting time of day.
        formatted.append(dic)
    return(formatted) 
all_formatted = format_articles(all_docs)
all_formatted[:5]
Explanation: This is all great, but it's pretty messy. What we’d really like to to have, eventually, is a CSV, with each row representing an article, and each column representing something about that article (header, date, etc). As we saw before, the best way to do this is to make a lsit of dictionaries, with each dictionary representing an article and each dictionary representing a field of metadata from that article (e.g. headline, date, etc.) We can do this with a custom function:
End of explanation
def format_articles(unformatted_docs):
    '''
    This function takes in a list of documents returned by the NYT api 
    and parses the documents into a list of dictionaries, 
    with 'id', 'header', 'date', 'lead paragrph' and 'word count' keys
    '''
    formatted = []
    for i in unformatted_docs:
        dic = {}
        dic['id'] = i['_id']
        dic['headline'] = i['headline']['main']
        dic['date'] = i['pub_date'][0:10] # cutting time of day.
        
        # YOUR CODE HERE
        
        formatted.append(dic)
    return(formatted) 
# uncomment to test
# all_formatted = format_articles(all_docs)
# all_formatted[:5]
Explanation: Challenge 4 Collect more fields
Edit the function above so that we include the lead_paragraph and word_count fields.
HINT: Some articles may not contain a lead_paragraph, in which case, it'll throw an error if you try to address this value (which doesn't exist.) You need to add a conditional statement that takes this into consideraiton. If
Advanced: Add another key that returns a list of keywords associated with the article.
End of explanation
keys = all_formatted[1]
# writing the rest
with open('all-formated.csv', 'w') as output_file:
    dict_writer = csv.DictWriter(output_file, keys)
    dict_writer.writeheader()
    dict_writer.writerows(all_formatted)
Explanation: 5. Exporting
We can now export the data to a CSV.
End of explanation
# YOUR CODE HERE
Explanation: Capstone Challenge
Using what you learned, tell me if Chris' claim (i.e. that Duke Ellington has gotten more popular lately) holds water.
End of explanation
<END_TASK>