<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 PDF is garbage
In this example, we are looking for a link to some source code 
Step1:  PDF is garbage, continued
If we remove line breaks to fix URLs that have been wrapped, we discover
that the visible line breaks in the document do not correspond to actual
line breaks in the represented text. The result is random garbage.
Step2:  Nope.
At this point, the author elects to flip a table. 
Let's try looking at the HTML version. I'll swipe some code from 
Dive into Python here, because 
finding URLs in a HTML document is what is known as a "Solved Problem."
Step3:  Here are all the URLs in the document...
Step4:  Bleh. That is mostly links in the references, ads and navigation cruft 
from the journal's content mismanagement system. Because their system
is heinously ad hoc, there is no base URL. So, we're forced to use an 
ad hoc exclusion list.
Step5:  Much better. Now, let's see if these exist...
Step6:  Looks like this will work, though we'll need to make a hand-curated list of
excluded URLs. Othersise, the counts of dead links could be badly skewed by
any issues within the journal's content mismanagement system, ad servers and
other irrelevent crud.
Walking through Zotero
Let's try walking through the publications in a Zotero library...
Step7:  So far so good. Let's have a look at the url attribute...
Step8:  Well, it looks like not all resources have URLs. Let's try looping over
some of these and extracting links...
Step9:  Clearly, we need to expand the excluded URL list. And we need to match 
domains, not URLs.
Step10:  This excluded list is getting sloppy as the author slowly lapses into 
a vegitative state, but we'll push on anyway.
Step11:  Some journals aggressivly ban and throttle IPs, so this process gets slow
and awful, but it works. Let's check these for dead links...
<ASSISTANT_TASK:>
Python Code:
urlre = re.compile( '(?P<url>https?://[^\s]+)' )
for page in doc :
    print urlre.findall( page )
Explanation: PDF is garbage
In this example, we are looking for a link to some source code :
http://prodege.jgi-psf.org//downloads/src
However, in the PDF, the URL is line wrapped, so the src is lost.
End of explanation
urlre = re.compile( '(?P<url>https?://[^\s]+)' )
for page in doc :
    print urlre.findall( page.replace('\n','') )
Explanation: PDF is garbage, continued
If we remove line breaks to fix URLs that have been wrapped, we discover
that the visible line breaks in the document do not correspond to actual
line breaks in the represented text. The result is random garbage.
End of explanation
from sgmllib import SGMLParser
class URLLister(SGMLParser):
    def reset(self):                              
        SGMLParser.reset(self)
        self.urls = []
    def start_a(self, attrs):                     
        href = [v for k, v in attrs if k=='href']  
        if href:
            self.urls.extend(href)
            
def get_urls_from(url):
    url_list = []
    import urllib
    usock = urllib.urlopen(url)
    parser = URLLister()
    parser.feed(usock.read())         
    usock.close()      
    parser.close()                    
    map(url_list.append, 
        [item for item in parser.urls if item.startswith(('http', 'ftp', 'www'))])
    return url_list
Explanation: Nope.
At this point, the author elects to flip a table. 
Let's try looking at the HTML version. I'll swipe some code from 
Dive into Python here, because 
finding URLs in a HTML document is what is known as a "Solved Problem."
End of explanation
urls = get_urls_from('http://www.nature.com/ismej/journal/v10/n1/full/ismej2015100a.html')
urls
Explanation: Here are all the URLs in the document...
End of explanation
excluded = [ 'http://www.nature.com',
             'http://dx.doi.org',
             'http://www.ncbi.nlm.nih.gov',
             'http://creativecommons.org',
             'https://s100.copyright.com',
             'http://mts-isme.nature.com',
             'http://www.isme-microbes.org',
             'http://ad.doubleclick.net',
             'http://mse.force.com',
             'http://links.isiglobalnet2.com',
             'http://www.readcube.com',
             'http://chemport.cas.org',
             'http://publicationethics.org/',
             'http://www.natureasia.com/'
           ]
def novel_url( url ) :
    for excluded_url in excluded :
        if url.startswith( excluded_url ) :
            return False
    return True
filter( novel_url, urls )
Explanation: Bleh. That is mostly links in the references, ads and navigation cruft 
from the journal's content mismanagement system. Because their system
is heinously ad hoc, there is no base URL. So, we're forced to use an 
ad hoc exclusion list.
End of explanation
import requests
for url in filter( novel_url, urls ) :
    request = requests.get( url )
    if request.status_code == 200:
        print 'Good : ', url
    else:
        print 'Fail : ', url
Explanation: Much better. Now, let's see if these exist...
End of explanation
from pyzotero import zotero
api_key    = open( 'zotero_api_key.txt' ).read().strip()
library_id = open( 'zotero_api_userID.txt' ).read().strip()
library_type = 'group'
group_id = '405341' # microBE.net group ID
zot = zotero.Zotero(group_id, library_type, api_key)
items = zot.top(limit=5)
# we've retrieved the latest five top-level items in our library
# we can print each item's item type and ID
for item in items:
    #print('Item: %s | Key: %s') % (item['data']['itemType'], item['data']['key'])
    print item['data']['key'], ':', item['data']['title']
Explanation: Looks like this will work, though we'll need to make a hand-curated list of
excluded URLs. Othersise, the counts of dead links could be badly skewed by
any issues within the journal's content mismanagement system, ad servers and
other irrelevent crud.
Walking through Zotero
Let's try walking through the publications in a Zotero library...
End of explanation
for item in items:
    print item['data']['key'], ':', item['data']['url']
Explanation: So far so good. Let's have a look at the url attribute...
End of explanation
for item in items:
    paper_url = item['data']['url']
    if paper_url.startswith( 'http' ) :
        link_urls = get_urls_from( paper_url )
        print item['data']['key']
        for url in filter( novel_url, link_urls ) :
            print '    ', url
Explanation: Well, it looks like not all resources have URLs. Let's try looping over
some of these and extracting links...
End of explanation
excluded = [ 'nature.com',
             'doi.org',
             'ncbi.nlm.nih.gov',
             'creativecommons.org',
             'copyright.com',
             'isme-microbes.org',
             'doubleclick.net',
             'force.com',
             'isiglobalnet2.com',
             'readcube.com',
             'cas.org',
             'publicationethics.org',
             'natureasia.com',
             'uq.edu.au',
             'edx.org',
             'facebook.com',
             'instagram.com',
             'youtube.com',
             'flickr.com',
             'twitter.com',
             'go8.edu.au',
             'google.com',
             'vimeo.com',
             'peerj.com',
             'mendeley.com',
             'cloudfront.net',
             'webofknowledge.com',
             'sciencedirect.com',
             'aol.com',
             'pinterest.com',
             'scopus.com',
             'live.com',
             'exlibrisgroup.com',
             'usyd.edu.au',
             'academicanalytics.com',
             'microbiomedigest.com',
             'ask.com',
             'sogou.com',
             'ou.com',
             'du.edu',
             'ru.nl',
             'freshdesk.com',
             'caltech.edu',
             'traackr.com',
             'adobe.com',
             'linkedin.com',
             'feedly.com',
             'google.co.uk',
             'glgoo.org',
             'library.wisc.edu',
             'lib.fsu.edu',
             'library.illinois.edu',
             'exchange.ou.edu',
             'lib.noaa.gov',
             'innocentive.com',
             'sfx.kcl.ac.uk',
             'sfx.unimi.it',
             'lib.utexas.edu',
             'orcid.org',
           ]
def novel_url( url ) :
    for excluded_url in excluded :
        if url.__contains__( excluded_url ) :
            return False
    return True
Explanation: Clearly, we need to expand the excluded URL list. And we need to match 
domains, not URLs.
End of explanation
for item in items:
    paper_url = item['data']['url']
    if paper_url.startswith( 'http' ) :
        try :
            link_urls = get_urls_from( paper_url )
            print item['data']['key']
            for url in list(set(filter( novel_url, link_urls ))) :
                print '    ', url
        except IOError :
            print item['data']['key'], 'FAILED'
Explanation: This excluded list is getting sloppy as the author slowly lapses into 
a vegitative state, but we'll push on anyway.
End of explanation
for item in items:
    paper_url = item['data']['url']
    if paper_url.startswith( 'http' ) :
        try :
            link_urls = get_urls_from( paper_url )
            print item['data']['key']
            for url in list(set(filter( novel_url, link_urls ))) :
                request = requests.get( url )
                if request.status_code == 200:
                    print '   Good : ', url
                else:
                    print '   Fail : ', url
        except IOError :
            print item['data']['key'], 'FAILED'
Explanation: Some journals aggressivly ban and throttle IPs, so this process gets slow
and awful, but it works. Let's check these for dead links...
End of explanation
<END_TASK>