<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 The Guardian API
In the beautiful_soup.ipynb notebook, I showed how BeautifulSoup can be used 
to parse messy HTML, tp extract information, and to act as a rudimentary web crawler. 
I used The Guardian as an illustrative example about how this can be achieved. 
The reason for choosing The Guardian was because they provide a REST API to their servers. 
With theise it is possible to perform specific queries on their servers, and to receive 
current information from their servers according to their API guide (ie in JSON)
http
Step1:  Inspect all sections and search for technology-based sections
Step2:  Manual query on whole API
Step3:  The API returns JSON, so we parse this using the in-built JSON library. 
The API specifies that all data are returned within the response key, even under failure. 
Thereofre, I have immediately descended to the response field 
Parsing the JSON
Step4:  Verifying the status code
It is important to verify that the status message is ok before continuing - if it is not ok no 'real' data 
will have been received.
Step5:  Listing the results
The API standard states that the results will be found in the results field under the response field. 
Furthermore, the URLs will be found in the webUrl field, and the title will be found in the webTitle 
field. 
First let's look to see what a single result looks like in full, and then I will print a restricted 
set of parameters on the full set of results .
Step6:  Let's now request a specific piece of content from the API. 
We select the ith result from the above response and get its apiUrl and id
Step7:  We then use the id to contstruct a search url string to request this piece of content from the API.
(Note that you need to include the api-key in the search, this is what I forgot in the lecture. You also need to specify if you want to include data fields other than the article metadata e.g. body and headline are included in the example below.)
Step8:  We can now do some simple text processing on the article text. e.g. count the word frequnecies
Step9:  So we have a dataframe with word occurence frequency in the article. 
But there is punctuation messing this up. For example, we see that again. appears once, as does providers,.  
One option to fix this would be to strip out the punctuation using Python string manipulation. But you could also use regular expressions to remove the punctuation. Below is a hacky example, but you can probably find a better solution.
Step10:  Note that the regex r'[^\w\s]' substitutes anything in body that is not a word \w or and blank space \s with the empty string ''.
<ASSISTANT_TASK:>
Python Code:
from __future__ import print_function
import requests 
import json 
Explanation: The Guardian API
In the beautiful_soup.ipynb notebook, I showed how BeautifulSoup can be used 
to parse messy HTML, tp extract information, and to act as a rudimentary web crawler. 
I used The Guardian as an illustrative example about how this can be achieved. 
The reason for choosing The Guardian was because they provide a REST API to their servers. 
With theise it is possible to perform specific queries on their servers, and to receive 
current information from their servers according to their API guide (ie in JSON)
http://open-platform.theguardian.com/
In order to use their API, you will need to register for an API key. 
At the time of writing (Feb 1, 2017) this was an automated process that can be completed at 
http://open-platform.theguardian.com/access/
The API is documented here: 
http://open-platform.theguardian.com/documentation/
and Python bindings to their API are provided by The Guardian here
https://github.com/prabhath6/theguardian-api-python
and these can easily be integrated into a web-crawler based on API calls, rather than being based 
on HTML parsing, etc. 
We use four parameters in our queries here: 
section: the section of the newspaper that we are interested in querying. In this case I'm lookin in 
the technology section 
order-by: I have specifie that the newest items should be closer to the front of the query list 
api-key: I have left this as test (which works here), but for real deployment of such a spider
a real API key should be specified 
page-size: The number of results to return.
End of explanation
url = 'https://content.guardianapis.com/sections?api-key=test'
req = requests.get(url)
src = req.text 
json.loads(src)['response']['status']
sections = json.loads(src)['response']
print(sections.keys())
print(json.dumps(sections['results'][0], indent=2, sort_keys=True))
for result in sections['results']: 
    if 'tech' in result['id'].lower(): 
        print(result['webTitle'], result['apiUrl'])
Explanation: Inspect all sections and search for technology-based sections
End of explanation
# Specify the arguments
args = {
    'section': 'technology', 
    'order-by': 'newest', 
    'api-key': 'test', 
    'page-size': '100',
    'q' : 'privacy%20AND%20data'
}
# Construct the URL
base_url = 'http://content.guardianapis.com/search'
url = '{}?{}'.format(
    base_url, 
    '&'.join(["{}={}".format(kk, vv) for kk, vv in args.items()])
)
# Make the request and extract the source
req = requests.get(url) 
src = req.text
print('Number of byes received:', len(src))
Explanation: Manual query on whole API
End of explanation
response = json.loads(src)['response']
print('The following are available:\n ', sorted(response.keys()))
Explanation: The API returns JSON, so we parse this using the in-built JSON library. 
The API specifies that all data are returned within the response key, even under failure. 
Thereofre, I have immediately descended to the response field 
Parsing the JSON
End of explanation
assert response['status'] == 'ok'
Explanation: Verifying the status code
It is important to verify that the status message is ok before continuing - if it is not ok no 'real' data 
will have been received.
End of explanation
print(json.dumps(response['results'][0], indent=2, sort_keys=True))
for result in response['results']: 
    print(result['webUrl'][:70], result['webTitle'][:20])
Explanation: Listing the results
The API standard states that the results will be found in the results field under the response field. 
Furthermore, the URLs will be found in the webUrl field, and the title will be found in the webTitle 
field. 
First let's look to see what a single result looks like in full, and then I will print a restricted 
set of parameters on the full set of results .
End of explanation
i = 0
api_url = response['results'][i]['apiUrl']
api_id = response['results'][i]['id']
print(api_url)
print(api_id)
Explanation: Let's now request a specific piece of content from the API. 
We select the ith result from the above response and get its apiUrl and id:
End of explanation
base_url = "https://content.guardianapis.com/search?"
search_string = "ids=%s&api-key=test&show-fields=headline,body" %api_id
url = base_url + search_string
print(url)
req = requests.get(url) 
src = req.text
response = json.loads(src)['response']
assert response['status'] == 'ok'
print(response['results'][0]['fields']['headline'])
body = response['results'][0]['fields']['body']
print(body)
Explanation: We then use the id to contstruct a search url string to request this piece of content from the API.
(Note that you need to include the api-key in the search, this is what I forgot in the lecture. You also need to specify if you want to include data fields other than the article metadata e.g. body and headline are included in the example below.)
End of explanation
words = body.replace('<p>','').replace('</p>','').split()
print(len(words))
unique_words = list(set(words))
print(len(unique_words))
#count_dictionary = {word: count for word, count in zip(words, [words.count(w) for w in words])}
count_dictionary = {'word': unique_words, 'count': [words.count(w) for w in unique_words]}
import pandas as pd
df = pd.DataFrame(count_dictionary)
df.sort_values(by='count', ascending=False)
Explanation: We can now do some simple text processing on the article text. e.g. count the word frequnecies:
End of explanation
import re  ## imports the regular expression library
words_wo_punctuation = re.sub(r'[^\w\s]','',body.replace('<p>','').replace('</p>','')).split()  
Explanation: So we have a dataframe with word occurence frequency in the article. 
But there is punctuation messing this up. For example, we see that again. appears once, as does providers,.  
One option to fix this would be to strip out the punctuation using Python string manipulation. But you could also use regular expressions to remove the punctuation. Below is a hacky example, but you can probably find a better solution.
End of explanation
unique_words = list(set(words_wo_punctuation))
print(len(unique_words))
count_dictionary = {'word': unique_words, 'count': [words_wo_punctuation.count(w) for w in unique_words]}
df = pd.DataFrame(count_dictionary)
df.sort_values(by='count', ascending=False)
Explanation: Note that the regex r'[^\w\s]' substitutes anything in body that is not a word \w or and blank space \s with the empty string ''.
End of explanation
<END_TASK>