<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 Source of the materials
Step1:  <span>Bio.Entrez</span> will then use this email address with each
call to Entrez. The <span>example.com</span> address is a reserved
domain name specifically for documentation (RFC 2606). Please DO NOT
use a random email – it’s better not to give an email at all. The
email parameter will be mandatory from June 1, 2010. In case of
excessive usage, NCBI will attempt to contact a user at the e-mail
address provided prior to blocking access to the E-utilities.
If you are using Biopython within some larger software suite, use
the tool parameter to specify this. You can either explicitly set
the tool name as a parameter with each call to Entrez (e.g. include
<span>tool=“MyLocalScript”</span> in the argument list), or you can
set a global tool name
Step2:  The tool parameter will default to Biopython.
For large queries, the NCBI also recommend using their session
    history feature (the WebEnv session cookie string,
    see Section History and WebEnv). This is only slightly
    more complicated.
In conclusion, be sensible with your usage levels. If you plan to
download lots of data, consider other options. For example, if you want
easy access to all the human genes, consider fetching each chromosome by
FTP as a GenBank file, and importing these into your own BioSQL database
(see Section [sec
Step3:  Since this is a fairly simple XML file, we could extract the information
it contains simply by string searching. Using Bio.Entrez’s parser
instead, we can directly parse this XML file into a Python object
Step4:  Now record is a dictionary with exactly one key
Step5:  The values stored in this key is the list of database names shown in the
XML above
Step6:  For each of these databases, we can use EInfo again to obtain more
information
Step7:  Try record["DbInfo"].keys() for other information stored in this
record. One of the most useful is a list of possible search fields for
use with ESearch
Step8:  That’s a long list, but indirectly this tells you that for the PubMed
database, you can do things like Jones[AUTH] to search the author
field, or Sanger[AFFL] to restrict to authors at the Sanger Centre.
This can be very handy - especially if you are not so familiar with a
particular database.
ESearch
Step9:  In this output, you see seven PubMed IDs (including 19304878 which is
the PMID for the Biopython application), which can be retrieved by
EFetch (see section EFetch
Step10:  Each of the IDs (126789333, 37222967, 37222966, …) is a GenBank
identifier. See section EFetch
Step11:  Again, we could use EFetch to obtain more information for each of these
journal IDs.
ESearch has many useful options — see the ESearch help
page
for more information.
EPost
Step12:  The returned XML includes two important strings, QueryKey and WebEnv
which together define your history session. You would extract these
values for use with another Entrez call such as EFetch
Step13:  Section History and WebEnv shows how to use the history feature.
ESummary
Step14:  EFetch
Step15:  The arguments rettype="gb" and retmode="text" let us download this
record in the GenBank format.
Note that until Easter 2009, the Entrez EFetch API let you use “genbank”
as the return type, however the NCBI now insist on using the official
return types of “gb” or “gbwithparts” (or “gp” for proteins) as
described on online. Also not that until Feb 2012, the Entrez EFetch API
would default to returning plain text files, but now defaults to XML.
Alternatively, you could for example use rettype="fasta" to get the
Fasta-format; see the EFetch Sequences Help
page
for other options. Remember – the available formats depend on which
database you are downloading from - see the main EFetch Help
page.
If you fetch the record in one of the formats accepted by Bio.SeqIO
(see Chapter [chapter
Step16:  Note that a more typical use would be to save the sequence data to a
local file, and then parse it with Bio.SeqIO. This can save you
having to re-download the same file repeatedly while working on your
script, and places less load on the NCBI’s servers. For example
Step17:  To get the output in XML format, which you can parse using the
Bio.Entrez.read() function, use retmode="xml"
Step18:  So, that dealt with sequences. For examples of parsing file formats
specific to the other databases (e.g. the MEDLINE format used in
PubMed), see Section Specialized parsers.
If you want to perform a search with Bio.Entrez.esearch(), and then
download the records with Bio.Entrez.efetch(), you should use the
WebEnv history feature – see Section History and WebEnv.
ELink
Step19:  The record variable consists of a Python list, one for each database
in which we searched. Since we specified only one PubMed ID to search
for, record contains only one item. This item is a dictionary
containing information about our search term, as well as all the related
items that were found
Step20:  The actual search results are stored as under the "Link" key. In
total, 110 items were found under standard search. Let’s now at the
first search result
Step21:  This is the article we searched for, which doesn’t help us much, so
let’s look at the second search result
Step22:  This paper, with PubMed ID 14630660, is about the Biopython PDB parser.
We can use a loop to print out all PubMed IDs
Step23:  Now that was nice, but personally I am often more interested to find out
if a paper has been cited. Well, ELink can do that too – at least for
journals in Pubmed Central (see Section [sec
Step24:  See the EGQuery help
page
for more information.
ESpell
Step25:  See the ESpell help
page
for more information. The main use of this is for GUI tools to provide
automatic suggestions for search terms.
Parsing huge Entrez XML files
The Entrez.read function reads the entire XML file returned by Entrez
into a single Python object, which is kept in memory. To parse Entrez
XML files too large to fit in memory, you can use the function
Entrez.parse. This is a generator function that reads records in the
XML file one by one. This function is only useful if the XML file
reflects a Python list object (in other words, if Entrez.read on a
computer with infinite memory resources would return a Python list).
For example, you can download the entire Entrez Gene database for a
given organism as a file from NCBI’s ftp site. These files can be very
large. As an example, on September 4, 2009, the file
Homo_sapiens.ags.gz, containing the Entrez Gene database for human,
had a size of 116576 kB. This file, which is in the ASN format, can be
converted into an XML file using NCBI’s gene2xml program (see NCBI’s
ftp site for more information)
Step26:  Here, the parser didn’t find the &lt;?xml ... tag with which an XML file
is supposed to start, and therefore decides (correctly) that the file is
not an XML file.
When your file is in the XML format but is corrupted (for example, by
ending prematurely), the parser will raise a CorruptedXMLError. Here is
an example of an XML file that ends prematurely
Step27:  Of course, the information contained in the XML tags that are not in the
DTD are not present in the record returned by Entrez.read.
Specialized parsers
The Bio.Entrez.read() function can parse most (if not all) XML output
returned by Entrez. Entrez typically allows you to retrieve records in
other formats, which may have some advantages compared to the XML format
in terms of readability (or download size).
To request a specific file format from Entrez using
Bio.Entrez.efetch() requires specifying the rettype and/or retmode
optional arguments. The different combinations are described for each
database type on the NCBI efetch
webpage.
One obvious case is you may prefer to download sequences in the FASTA or
GenBank/GenPept plain text formats (which can then be parsed with
Bio.SeqIO, see Sections [sec
Step28:  The record now contains the Medline record as a Python dictionary
Step29:  The key names used in a Medline record can be rather obscure; use
Step30:  for a brief summary.
To parse a file containing multiple Medline records, you can use the
parse function instead
Step31:  Instead of parsing Medline records stored in files, you can also parse
Medline records downloaded by Bio.Entrez.efetch. For example, let’s
look at all Medline records in PubMed related to Biopython
Step32:  We now use Bio.Entrez.efetch to download these Medline records
Step33:  Here, we specify rettype="medline", retmode="text" to obtain the
Medline records in plain-text Medline format. Now we use Bio.Medline
to parse these records
Step34:  For comparison, here we show an example using the XML format
Step35:  Note that in both of these examples, for simplicity we have naively
combined ESearch and EFetch. In this situation, the NCBI would expect
you to use their history feature, as illustrated in
Section History and WebEnv.
Parsing GEO records
GEO (Gene Expression Omnibus) is a
data repository of high-throughput gene expression and hybridization
array data. The Bio.Geo module can be used to parse GEO-formatted
data.
The following code fragment shows how to parse the example GEO file
GSE16.txt into a record and print the record
Step36:  You can search the “gds” database (GEO datasets) with ESearch
Step37:  From the Entrez website, UID “200000016” is GDS16 while the other hit
“100000028” is for the associated platform, GPL28. Unfortunately, at the
time of writing the NCBI don’t seem to support downloading GEO files
using Entrez (not as XML, nor in the Simple Omnibus Format in Text
(SOFT) format).
However, it is actually pretty straight forward to download the GEO
files by FTP or HTTP from http
Step38:  The record returned by UniGene.read is a Python object with
attributes corresponding to the fields in the UniGene record. For
example,
Step39:  The EXPRESS and RESTR_EXPR lines are stored as Python lists of
strings
Step40:  and similarly for the PROTSIM and SEQUENCE lines.
To parse a file containing more than one UniGene record, use the parse
function in Bio.UniGene
Step41:  Using a proxy
Normally you won’t have to worry about using a proxy, but if this is an
issue on your network here is how to deal with it. Internally,
Bio.Entrez uses the standard Python library urllib for accessing the
NCBI servers. This will check an environment variable called
http_proxy to configure any simple proxy automatically. Unfortunately
this module does not support the use of proxies which require
authentication.
You may choose to set the http_proxy environment variable once (how
you do this will depend on your operating system). Alternatively you can
set this within Python at the start of your script, for example
Step42:  Now we use the Bio.Entrez.efetch function to download the PubMed IDs
of these 463 articles
Step43:  Now that we’ve got them, we obviously want to get the corresponding
Medline records and extract the information from them. Here, we’ll
download the Medline records in the Medline flat-file format, and use
the Bio.Medline module to parse them
Step44:  NOTE - We’ve just done a separate search and fetch here, the NCBI much
prefer you to take advantage of their history support in this situation.
See Section History and WebEnv.
Keep in mind that records is an iterator, so you can iterate through
the records only once. If you want to save the records, you can convert
them to a list
Step45:  Let’s now iterate over the records to print out some information about
each record
Step46:  The output for this looks like
Step47:  Hopefully this section gave you an idea of the power and flexibility of
the Entrez and Medline interfaces and how they can be used together.
Searching, downloading, and parsing Entrez Nucleotide records {#subsec
Step48:  So, we expect to find 814 Entrez Nucleotide records (this is the number
I obtained in 2008; it is likely to increase in the future). If you find
some ridiculously high number of hits, you may want to reconsider if you
really want to download all of them, which is our next step
Step49:  Here, record is a Python dictionary containing the search results and
some auxiliary information. Just for information, let’s look at what is
stored in this dictionary
Step50:  First, let’s check how many results were found
Step51:  which is the number we expected. The 814 results are stored in
record['IdList']
Step52:  Let’s look at the first five results
Step53:  [sec
Step54:  Each of these records corresponds to one GenBank record.
Step55:  You could use this to quickly set up searches – but for heavy usage, see
Section History and WebEnv.
Searching, downloading, and parsing GenBank records {#sec
Step56:  Now we download the list of GenBank identifiers
Step57:  Now we use these GIs to download the GenBank records - note that with
older versions of Biopython you had to supply a comma separated list of
GI numbers to Entrez, as of Biopython 1.59 you can pass a list and this
is converted for you
Step58:  If you want to look at the raw GenBank files, you can read from this
handle and print out the result
Step59:  In this case, we are just getting the raw records. To get the records in
a more Python-friendly form, we can use Bio.SeqIO to parse the GenBank
data into SeqRecord objects, including SeqFeature objects (see
Chapter [chapter
Step60:  We can now step through the records and look at the information we are
interested in
Step61:  Using these automated query retrieval functionality is a big plus over
doing things by hand. Although the module should obey the NCBI’s max
three queries per second rule, the NCBI have other recommendations like
avoiding peak hours. See Section [sec
Step62:  Now, we use efetch to download this entry in the Taxonomy database,
and then parse it
Step63:  Again, this record stores lots of information
Step64:  We can get the lineage directly from this record
Step65:  The record data contains much more than just the information shown here
- for example look under LineageEx instead of Lineage and you’ll get
the NCBI taxon identifiers of the lineage entries too.
Using the history and WebEnv
Often you will want to make a series of linked queries. Most typically,
running a search, perhaps refining the search, and then retrieving
detailed search results. You can do this by making a series of
separate calls to Entrez. However, the NCBI prefer you to take advantage
of their history support - for example combining ESearch and EFetch.
Another typical use of the history support would be to combine EPost and
EFetch. You use EPost to upload a list of identifiers, which starts a
new history session. You then download the records with EFetch by
referring to the session (instead of the identifiers).
Searching for and downloading sequences using the history
Suppose we want to search and download all the Opuntia rpl16
nucleotide sequences, and store them in a FASTA file. As shown in
Section [sec
Step66:  When you get the XML output back, it will still include the usual search
results. However, you also get given two additional pieces of information, the
WebEnv session cookie, and the QueryKey
Step67:  Having stored these values in variables <span>session_cookie</span> and
<span>query_key</span> we can use them as parameters to
Bio.Entrez.efetch() instead of giving the GI numbers as identifiers.
While for small searches you might be OK downloading everything at once,
it is better to download in batches. You use the <span>retstart</span>
and <span>retmax</span> parameters to specify which range of search
results you want returned (starting entry using zero-based counting, and
maximum number of results to return). Sometimes you will get
intermittent errors from Entrez, HTTPError 5XX, we use a try except
pause retry block to address this. For example,
```
from Bio import Entrez
import time
try
Step68:  Great - eleven articles. But why hasn’t the Biopython application note
been found (PubMed ID 19304878)? Well, as you might have guessed from
the variable names, there are not actually PubMed IDs, but PubMed
Central IDs. Our application note is the third citing paper in that
list, PMCID 2682512.
So, what if (like me) you’d rather get back a list of PubMed IDs? Well
we can call ELink again to translate them. This becomes a two step
process, so by now you should expect to use the history feature to
accomplish it (Section History and WebEnv).
But first, taking the more straightforward approach of making a second
(separate) call to ELink
<ASSISTANT_TASK:>
Python Code:
from Bio import Entrez
Entrez.email = "A.N.Other@example.com"
Explanation: Source of the materials: Biopython cookbook (adapted)
<font color='red'>Status: Draft</font>
Accessing NCBI’s Entrez databases
Entrez Guidelines
EInfo: Obtaining information about the Entrez databases
ESearch: Searching the Entrez databases
EPost: Uploading a list of identifiers
EFetch: Downloading full records from Entrez
History and WebEnv
Specialized parsers
Examples
Entrez (http://www.ncbi.nlm.nih.gov/Entrez) is a data retrieval system
that provides users access to NCBI’s databases such as PubMed, GenBank,
GEO, and many others. You can access Entrez from a web browser to
manually enter queries, or you can use Biopython’s Bio.Entrez module
for programmatic access to Entrez. The latter allows you for example to
search PubMed or download GenBank records from within a Python script.
The Bio.Entrez module makes use of the Entrez Programming Utilities
(also known as EUtils), consisting of eight tools that are described in
detail on NCBI’s page at http://www.ncbi.nlm.nih.gov/entrez/utils/.
Each of these tools corresponds to one Python function in the
Bio.Entrez module, as described in the sections below. This module
makes sure that the correct URL is used for the queries, and that not
more than one request is made every three seconds, as required by NCBI.
The output returned by the Entrez Programming Utilities is typically in
XML format. To parse such output, you have several options:
Use Bio.Entrez’s parser to parse the XML output into a Python
    object;
Use the DOM (Document Object Model) parser in Python’s standard
    library;
Use the SAX (Simple API for XML) parser in Python’s standard
    library;
Read the XML output as raw text, and parse it by string searching
    and manipulation.
For the DOM and SAX parsers, see the Python documentation. The parser in
Bio.Entrez is discussed below.
NCBI uses DTD (Document Type Definition) files to describe the structure
of the information contained in XML files. Most of the DTD files used by
NCBI are included in the Biopython distribution. The Bio.Entrez parser
makes use of the DTD files when parsing an XML file returned by NCBI
Entrez.
Occasionally, you may find that the DTD file associated with a specific
XML file is missing in the Biopython distribution. In particular, this
may happen when NCBI updates its DTD files. If this happens,
Entrez.read will show a warning message with the name and URL of the
missing DTD file. The parser will proceed to access the missing DTD file
through the internet, allowing the parsing of the XML file to continue.
However, the parser is much faster if the DTD file is available locally.
For this purpose, please download the DTD file from the URL in the
warning message and place it in the directory
...site-packages/Bio/Entrez/DTDs, containing the other DTD files. If
you don’t have write access to this directory, you can also place the
DTD file in ~/.biopython/Bio/Entrez/DTDs, where ~ represents your
home directory. Since this directory is read before the directory
...site-packages/Bio/Entrez/DTDs, you can also put newer versions of
DTD files there if the ones in ...site-packages/Bio/Entrez/DTDs become
outdated. Alternatively, if you installed Biopython from source, you can
add the DTD file to the source code’s Bio/Entrez/DTDs directory, and
reinstall Biopython. This will install the new DTD file in the correct
location together with the other DTD files.
The Entrez Programming Utilities can also generate output in other
formats, such as the Fasta or GenBank file formats for sequence
databases, or the MedLine format for the literature database, discussed
in Section Specialized parsers.
Entrez Guidelines
Before using Biopython to access the NCBI’s online resources (via
Bio.Entrez or some of the other modules), please read the NCBI’s Entrez User Requirements.
If the NCBI finds you are abusing their systems, they can and will ban
your access!
To paraphrase:
For any series of more than 100 requests, do this at weekends or
    outside USA peak times. This is up to you to obey.
Use the http://eutils.ncbi.nlm.nih.gov address, not the standard
    NCBI Web address. Biopython uses this web address.
Make no more than three requests every seconds (relaxed from at most
    one request every three seconds in early 2009). This is
    automatically enforced by Biopython.
Use the optional email parameter so the NCBI can contact you if
    there is a problem. You can either explicitly set this as a
    parameter with each call to Entrez (e.g. include
    <span>email=“A.N.Other@example.com”</span> in the argument list), or
    you can set a global email address:
End of explanation
from Bio import Entrez
Entrez.tool = "MyLocalScript"
Explanation: <span>Bio.Entrez</span> will then use this email address with each
call to Entrez. The <span>example.com</span> address is a reserved
domain name specifically for documentation (RFC 2606). Please DO NOT
use a random email – it’s better not to give an email at all. The
email parameter will be mandatory from June 1, 2010. In case of
excessive usage, NCBI will attempt to contact a user at the e-mail
address provided prior to blocking access to the E-utilities.
If you are using Biopython within some larger software suite, use
the tool parameter to specify this. You can either explicitly set
the tool name as a parameter with each call to Entrez (e.g. include
<span>tool=“MyLocalScript”</span> in the argument list), or you can
set a global tool name:
End of explanation
from Bio import Entrez
Entrez.email = "A.N.Other@example.com"     # Always tell NCBI who you are
handle = Entrez.einfo()
result = handle.read()
print(result)
Explanation: The tool parameter will default to Biopython.
For large queries, the NCBI also recommend using their session
    history feature (the WebEnv session cookie string,
    see Section History and WebEnv). This is only slightly
    more complicated.
In conclusion, be sensible with your usage levels. If you plan to
download lots of data, consider other options. For example, if you want
easy access to all the human genes, consider fetching each chromosome by
FTP as a GenBank file, and importing these into your own BioSQL database
(see Section [sec:BioSQL]).
EInfo: Obtaining information about the Entrez databases
einfo source
EInfo provides field index term counts, last update, and available links
for each of NCBI’s databases. In addition, you can use EInfo to obtain a
list of all database names accessible through the Entrez utilities. The variable result now contains a list of databases in XML format:
End of explanation
from Bio import Entrez
handle = Entrez.einfo()
record = Entrez.read(handle)
Explanation: Since this is a fairly simple XML file, we could extract the information
it contains simply by string searching. Using Bio.Entrez’s parser
instead, we can directly parse this XML file into a Python object:
End of explanation
record.keys()
Explanation: Now record is a dictionary with exactly one key:
End of explanation
record["DbList"]
Explanation: The values stored in this key is the list of database names shown in the
XML above:
End of explanation
from Bio import Entrez
handle = Entrez.einfo(db="pubmed")
record = Entrez.read(handle)
record["DbInfo"]["Description"]
record['DbInfo'].keys()
handle = Entrez.einfo(db="pubmed")
record = Entrez.read(handle)
record["DbInfo"]["Description"]
record["DbInfo"]["Count"]
record["DbInfo"]["LastUpdate"]
Explanation: For each of these databases, we can use EInfo again to obtain more
information:
End of explanation
for field in record["DbInfo"]["FieldList"]:
    print("%(Name)s, %(FullName)s, %(Description)s" % field)
Explanation: Try record["DbInfo"].keys() for other information stored in this
record. One of the most useful is a list of possible search fields for
use with ESearch:
End of explanation
from Bio import Entrez
Entrez.email = "A.N.Other@example.com"     # Always tell NCBI who you are
handle = Entrez.esearch(db="pubmed", term="biopython")
record = Entrez.read(handle)
record["IdList"]
record
Explanation: That’s a long list, but indirectly this tells you that for the PubMed
database, you can do things like Jones[AUTH] to search the author
field, or Sanger[AFFL] to restrict to authors at the Sanger Centre.
This can be very handy - especially if you are not so familiar with a
particular database.
ESearch: Searching the Entrez databases
To search any of these databases, we use Bio.Entrez.esearch(). For
example, let’s search in PubMed for publications related to Biopython:
End of explanation
handle = Entrez.esearch(db="nucleotide", term="Cypripedioideae[Orgn] AND matK[Gene]")
record = Entrez.read(handle)
record["Count"]
record["IdList"]
Explanation: In this output, you see seven PubMed IDs (including 19304878 which is
the PMID for the Biopython application), which can be retrieved by
EFetch (see section EFetch: Downloading full records from Entrez).
You can also use ESearch to search GenBank. Here we’ll do a quick search
for the matK gene in Cypripedioideae orchids (see
Section [sec:entrez-einfo] about EInfo for one way to find out which
fields you can search in each Entrez database):
End of explanation
# nlmcatalog
# handle = Entrez.esearch(db="nlmcatalog", term="computational")
# record = Entrez.read(handle)
# record["Count"]
handle = Entrez.esearch(db="nlmcatalog", term="biopython[Journal]", RetMax='20')
record = Entrez.read(handle)
print("{} computational Journals found".format(record["Count"]))
print("The first 20 are\n{}".format(record['IdList']))
Explanation: Each of the IDs (126789333, 37222967, 37222966, …) is a GenBank
identifier. See section EFetch: Downloading full records from Entrez for information on how to
actually download these GenBank records.
Note that instead of a species name like Cypripedioideae[Orgn], you
can restrict the search using an NCBI taxon identifier, here this would
be txid158330[Orgn]. This isn’t currently documented on the ESearch
help page - the NCBI explained this in reply to an email query. You can
often deduce the search term formatting by playing with the Entrez web
interface. For example, including complete[prop] in a genome search
restricts to just completed genomes.
As a final example, let’s get a list of computational journal titles:
End of explanation
from Bio import Entrez
Entrez.email = "A.N.Other@example.com"     # Always tell NCBI who you are
id_list = ["19304878", "18606172", "16403221", "16377612", "14871861", "14630660"]
print(Entrez.epost("pubmed", id=",".join(id_list)).read())
Explanation: Again, we could use EFetch to obtain more information for each of these
journal IDs.
ESearch has many useful options — see the ESearch help
page
for more information.
EPost: Uploading a list of identifiers
EPost uploads a list of UIs for use in subsequent search strategies; see
the EPost help
page
for more information. It is available from Biopython through the
Bio.Entrez.epost() function.
To give an example of when this is useful, suppose you have a long list
of IDs you want to download using EFetch (maybe sequences, maybe
citations – anything). When you make a request with EFetch your list of
IDs, the database etc, are all turned into a long URL sent to the
server. If your list of IDs is long, this URL gets long, and long URLs
can break (e.g. some proxies don’t cope well).
Instead, you can break this up into two steps, first uploading the list
of IDs using EPost (this uses an “HTML post” internally, rather than an
“HTML get”, getting round the long URL problem). With the history
support, you can then refer to this long list of IDs, and download the
associated data with EFetch.
Let’s look at a simple example to see how EPost works – uploading some
PubMed identifiers:
End of explanation
from Bio import Entrez
Entrez.email = "A.N.Other@example.com"     # Always tell NCBI who you are
id_list = ["19304878", "18606172", "16403221", "16377612", "14871861", "14630660"]
search_results = Entrez.read(Entrez.epost("pubmed", id=",".join(id_list)))
webenv = search_results["WebEnv"]
query_key = search_results["QueryKey"]
Explanation: The returned XML includes two important strings, QueryKey and WebEnv
which together define your history session. You would extract these
values for use with another Entrez call such as EFetch:
End of explanation
from Bio import Entrez
Entrez.email = "A.N.Other@example.com"     # Always tell NCBI who you are
handle = Entrez.esummary(db="nlmcatalog", term="[journal]", id="101660833")
record = Entrez.read(handle)
info = record[0]['TitleMainList'][0]
print("Journal info\nid: {}\nTitle: {}".format(record[0]["Id"], info["Title"]))
Explanation: Section History and WebEnv shows how to use the history feature.
ESummary: Retrieving summaries from primary IDs
ESummary retrieves document summaries from a list of primary IDs (see
the ESummary help
page
for more information). In Biopython, ESummary is available as
Bio.Entrez.esummary(). Using the search result above, we can for
example find out more about the journal with ID 30367:
End of explanation
from Bio import Entrez
Entrez.email = "A.N.Other@example.com"     # Always tell NCBI who you are
handle = Entrez.efetch(db="nucleotide", id="186972394", rettype="gb", retmode="text")
print(handle.read())
Explanation: EFetch: Downloading full records from Entrez
EFetch is what you use when you want to retrieve a full record from
Entrez. This covers several possible databases, as described on the main
EFetch Help
page.
For most of their databases, the NCBI support several different file
formats. Requesting a specific file format from Entrez using
Bio.Entrez.efetch() requires specifying the rettype and/or retmode
optional arguments. The different combinations are described for each
database type on the pages linked to on NCBI efetch
webpage
(e.g.
literature,
sequences
and
taxonomy).
One common usage is downloading sequences in the FASTA or
GenBank/GenPept plain text formats (which can then be parsed with
Bio.SeqIO, see Sections [sec:SeqIO_GenBank_Online]
and EFetch: Downloading full records from Entrez). From the Cypripedioideae example above, we can
download GenBank record 186972394 using Bio.Entrez.efetch:
End of explanation
from Bio import Entrez, SeqIO
handle = Entrez.efetch(db="nucleotide", id="186972394", rettype="gb", retmode="text")
record = SeqIO.read(handle, "genbank")
handle.close()
print(record)
Explanation: The arguments rettype="gb" and retmode="text" let us download this
record in the GenBank format.
Note that until Easter 2009, the Entrez EFetch API let you use “genbank”
as the return type, however the NCBI now insist on using the official
return types of “gb” or “gbwithparts” (or “gp” for proteins) as
described on online. Also not that until Feb 2012, the Entrez EFetch API
would default to returning plain text files, but now defaults to XML.
Alternatively, you could for example use rettype="fasta" to get the
Fasta-format; see the EFetch Sequences Help
page
for other options. Remember – the available formats depend on which
database you are downloading from - see the main EFetch Help
page.
If you fetch the record in one of the formats accepted by Bio.SeqIO
(see Chapter [chapter:Bio.SeqIO]), you could directly parse it into a
SeqRecord:
End of explanation
import os
from Bio import SeqIO
from Bio import Entrez
Entrez.email = "A.N.Other@example.com"     # Always tell NCBI who you are
filename = "gi_186972394.gbk"
if not os.path.isfile(filename):
    # Downloading...
    with Entrez.efetch(db="nucleotide",id="186972394",rettype="gb", retmode="text") as net_handle:
        with open(filename, "w") as out_handle:
            out_handle.write(net_handle.read())
        print("Saved")
print("Parsing...")
record = SeqIO.read(filename, "genbank")
print(record)
Explanation: Note that a more typical use would be to save the sequence data to a
local file, and then parse it with Bio.SeqIO. This can save you
having to re-download the same file repeatedly while working on your
script, and places less load on the NCBI’s servers. For example:
End of explanation
from Bio import Entrez
handle = Entrez.efetch(db="nucleotide", id="186972394", retmode="xml")
record = Entrez.read(handle)
handle.close()
record[0]["GBSeq_definition"]
record[0]["GBSeq_source"]
Explanation: To get the output in XML format, which you can parse using the
Bio.Entrez.read() function, use retmode="xml":
End of explanation
from Bio import Entrez
Entrez.email = "A.N.Other@example.com"
pmid = "19304878"
record = Entrez.read(Entrez.elink(dbfrom="pubmed", id=pmid))
print(record[0].keys())
print('The record is from the {} database.'.format(record[0]["DbFrom"]))
print('The IdList is {}.'.format(record[0]["IdList"]))
Explanation: So, that dealt with sequences. For examples of parsing file formats
specific to the other databases (e.g. the MEDLINE format used in
PubMed), see Section Specialized parsers.
If you want to perform a search with Bio.Entrez.esearch(), and then
download the records with Bio.Entrez.efetch(), you should use the
WebEnv history feature – see Section History and WebEnv.
ELink: Searching for related items in NCBI Entrez
ELink, available from Biopython as Bio.Entrez.elink(), can be used to
find related items in the NCBI Entrez databases. For example, you can us
this to find nucleotide entries for an entry in the gene database, and
other cool stuff.
Let’s use ELink to find articles related to the Biopython application
note published in Bioinformatics in 2009. The PubMed ID of this
article is 19304878:
End of explanation
print('There are {} search results'.format(len(record[0]["LinkSetDb"])))
for linksetdb in record[0]["LinkSetDb"]:
    print(linksetdb["DbTo"], linksetdb["LinkName"], len(linksetdb["Link"]))
Explanation: The record variable consists of a Python list, one for each database
in which we searched. Since we specified only one PubMed ID to search
for, record contains only one item. This item is a dictionary
containing information about our search term, as well as all the related
items that were found:
The "LinkSetDb" key contains the search results, stored as a list
consisting of one item for each target database. In our search results,
we only find hits in the PubMed database (although sub-divided into
categories):
End of explanation
record[0]["LinkSetDb"][0]["Link"][0]
Explanation: The actual search results are stored as under the "Link" key. In
total, 110 items were found under standard search. Let’s now at the
first search result:
End of explanation
record[0]["LinkSetDb"][0]["Link"][1]
Explanation: This is the article we searched for, which doesn’t help us much, so
let’s look at the second search result:
End of explanation
for link in record[0]["LinkSetDb"][0]["Link"]:
    print(link["Id"])
Explanation: This paper, with PubMed ID 14630660, is about the Biopython PDB parser.
We can use a loop to print out all PubMed IDs:
End of explanation
from Bio import Entrez
Entrez.email = "A.N.Other@example.com"     # Always tell NCBI who you are
handle = Entrez.egquery(term="biopython")
record = Entrez.read(handle)
for row in record["eGQueryResult"]:
    print(row["DbName"], row["Count"])
Explanation: Now that was nice, but personally I am often more interested to find out
if a paper has been cited. Well, ELink can do that too – at least for
journals in Pubmed Central (see Section [sec:elink-citations]).
For help on ELink, see the ELink help
page.
There is an entire sub-page just for the link
names,
describing how different databases can be cross referenced.
EGQuery: Global Query - counts for search terms
EGQuery provides counts for a search term in each of the Entrez
databases (i.e. a global query). This is particularly useful to find out
how many items your search terms would find in each database without
actually performing lots of separate searches with ESearch (see the
example in [subsec:entrez_example_genbank] below).
In this example, we use Bio.Entrez.egquery() to obtain the counts for
“Biopython”:
End of explanation
from Bio import Entrez
Entrez.email = "A.N.Other@example.com"     # Always tell NCBI who you are
handle = Entrez.espell(term="biopythooon")
record = Entrez.read(handle)
record["Query"]
record["CorrectedQuery"]
Explanation: See the EGQuery help
page
for more information.
ESpell: Obtaining spelling suggestions
ESpell retrieves spelling suggestions. In this example, we use
Bio.Entrez.espell() to obtain the correct spelling of Biopython:
End of explanation
from Bio import Entrez
from Bio.Entrez.Parser import NotXMLError
handle = open("data/NC_005816.fna", 'rb') # a Fasta file
try:
    record = Entrez.read(handle)
except NotXMLError as e:
    print('We are expecting to get NotXMLError')
    print(e)
Explanation: See the ESpell help
page
for more information. The main use of this is for GUI tools to provide
automatic suggestions for search terms.
Parsing huge Entrez XML files
The Entrez.read function reads the entire XML file returned by Entrez
into a single Python object, which is kept in memory. To parse Entrez
XML files too large to fit in memory, you can use the function
Entrez.parse. This is a generator function that reads records in the
XML file one by one. This function is only useful if the XML file
reflects a Python list object (in other words, if Entrez.read on a
computer with infinite memory resources would return a Python list).
For example, you can download the entire Entrez Gene database for a
given organism as a file from NCBI’s ftp site. These files can be very
large. As an example, on September 4, 2009, the file
Homo_sapiens.ags.gz, containing the Entrez Gene database for human,
had a size of 116576 kB. This file, which is in the ASN format, can be
converted into an XML file using NCBI’s gene2xml program (see NCBI’s
ftp site for more information):
```
gene2xml -b T -i Homo_sapiens.ags -o Homo_sapiens.xml
```
The resulting XML file has a size of 6.1 GB. Attempting Entrez.read on
this file will result in a MemoryError on many computers.
The XML file Homo_sapiens.xml consists of a list of Entrez gene
records, each corresponding to one Entrez gene in human. Entrez.parse
retrieves these gene records one by one. You can then print out or store
the relevant information in each record by iterating over the records.
For example, this script iterates over the Entrez gene records and
prints out the gene numbers and names for all current genes:
<font color='red'>TODO: need alternate example, download option or ...</font>
python
from Bio import Entrez
handle = open("Homo_sapiens.xml")
records = Entrez.parse(handle)
python
for record in records:
    status = record['Entrezgene_track-info']['Gene-track']['Gene-track_status']
    if status.attributes['value']=='discontinued':
        continue
    geneid = record['Entrezgene_track-info']['Gene-track']['Gene-track_geneid']
    genename = record['Entrezgene_gene']['Gene-ref']['Gene-ref_locus']
    print(geneid, genename)
This will print:
```
1 A1BG
2 A2M
3 A2MP
8 AA
9 NAT1
10 NAT2
11 AACP
12 SERPINA3
13 AADAC
14 AAMP
15 AANAT
16 AARS
17 AAVS1
...
```
Handling errors
Three things can go wrong when parsing an XML file:
The file may not be an XML file to begin with;
The file may end prematurely or otherwise be corrupted;
The file may be correct XML, but contain items that are not
    represented in the associated DTD.
The first case occurs if, for example, you try to parse a Fasta file as
if it were an XML file:
End of explanation
from Bio import Entrez
handle = open("data/einfo3.xml", 'rb')
record = Entrez.read(handle, validate=False)
Explanation: Here, the parser didn’t find the &lt;?xml ... tag with which an XML file
is supposed to start, and therefore decides (correctly) that the file is
not an XML file.
When your file is in the XML format but is corrupted (for example, by
ending prematurely), the parser will raise a CorruptedXMLError. Here is
an example of an XML file that ends prematurely:
```xml
<?xml version="1.0"?>
<!DOCTYPE eInfoResult PUBLIC "-//NLM//DTD eInfoResult, 11 May 2002//EN" "http://www.ncbi.nlm.nih.gov/entrez/query/DTD/eInfo_020511.dtd">
<eInfoResult>
<DbList>
        <DbName>pubmed</DbName>
        <DbName>protein</DbName>
        <DbName>nucleotide</DbName>
        <DbName>nuccore</DbName>
        <DbName>nucgss</DbName>
        <DbName>nucest</DbName>
        <DbName>structure</DbName>
        <DbName>genome</DbName>
        <DbName>books</DbName>
        <DbName>cancerchromosomes</DbName>
        <DbName>cdd</DbName>
```
which will generate the following traceback:
```python
ExpatError                                Traceback (most recent call last)
/Users/vincentdavis/anaconda/envs/py35/lib/python3.5/site-packages/Bio/Entrez/Parser.py in read(self, handle)
    214         try:
--> 215             self.parser.ParseFile(handle)
    216         except expat.ExpatError as e:
ExpatError: syntax error: line 1, column 0
During handling of the above exception, another exception occurred:
NotXMLError                               Traceback (most recent call last)
<ipython-input-63-ac0523d72453> in <module>()
----> 1 Entrez.read(handle)
/Users/vincentdavis/anaconda/envs/py35/lib/python3.5/site-packages/Bio/Entrez/init.py in read(handle, validate)
    419     from .Parser import DataHandler
    420     handler = DataHandler(validate)
--> 421     record = handler.read(handle)
    422     return record
    423 
/Users/vincentdavis/anaconda/envs/py35/lib/python3.5/site-packages/Bio/Entrez/Parser.py in read(self, handle)
    223                 # We have not seen the initial <!xml declaration, so probably
    224                 # the input data is not in XML format.
--> 225                 raise NotXMLError(e)
    226         try:
    227             return self.object
NotXMLError: Failed to parse the XML data (syntax error: line 1, column 0). Please make sure that the input data are in XML format.
```
Note that the error message tells you at what point in the XML file the
error was detected.
The third type of error occurs if the XML file contains tags that do not
have a description in the corresponding DTD file. This is an example of
such an XML file:
```xml
<?xml version="1.0"?>
<!DOCTYPE eInfoResult PUBLIC "-//NLM//DTD eInfoResult, 11 May 2002//EN" "http://www.ncbi.nlm.nih.gov/entrez/query/DTD/eInfo_020511.dtd">
<eInfoResult>
        <DbInfo>
        <DbName>pubmed</DbName>
        <MenuName>PubMed</MenuName>
        <Description>PubMed bibliographic record</Description>
        <Count>20161961</Count>
        <LastUpdate>2010/09/10 04:52</LastUpdate>
        <FieldList>
                <Field>
...
                </Field>
        </FieldList>
        <DocsumList>
                <Docsum>
                        <DsName>PubDate</DsName>
                        <DsType>4</DsType>
                        <DsTypeName>string</DsTypeName>
                </Docsum>
                <Docsum>
                        <DsName>EPubDate</DsName>
...
        </DbInfo>
</eInfoResult>
```
In this file, for some reason the tag &lt;DocsumList&gt; (and several
others) are not listed in the DTD file eInfo_020511.dtd, which is
specified on the second line as the DTD for this XML file. By default,
the parser will stop and raise a ValidationError if it cannot find some
tag in the DTD:
python
from Bio import Entrez
handle = open("data/einfo3.xml", 'rb')
record = Entrez.read(handle)
```python
ValidationError                           Traceback (most recent call last)
<ipython-input-65-cfb96ec3d2ca> in <module>()
      1 from Bio import Entrez
      2 handle = open("data/einfo3.xml", 'rb')
----> 3 record = Entrez.read(handle)
/Users/vincentdavis/anaconda/envs/py35/lib/python3.5/site-packages/Bio/Entrez/init.py in read(handle, validate)
    419     from .Parser import DataHandler
    420     handler = DataHandler(validate)
--> 421     record = handler.read(handle)
    422     return record
    423 
/Users/vincentdavis/anaconda/envs/py35/lib/python3.5/site-packages/Bio/Entrez/Parser.py in read(self, handle)
    213             raise IOError("Can't parse a closed handle")
    214         try:
--> 215             self.parser.ParseFile(handle)
    216         except expat.ExpatError as e:
    217             if self.parser.StartElementHandler:
-------src-dir--------/Python-3.5.1/Modules/pyexpat.c in StartElement()
/Users/vincentdavis/anaconda/envs/py35/lib/python3.5/site-packages/Bio/Entrez/Parser.py in startElementHandler(self, name, attrs)
    348             # Element not found in DTD
    349             if self.validating:
--> 350                 raise ValidationError(name)
    351             else:
    352                 # this will not be stored in the record
ValidationError: Failed to find tag 'DocsumList' in the DTD. To skip all tags that are not represented in the DTD, please call Bio.Entrez.read or Bio.Entrez.parse with validate=False.
```
Optionally, you can instruct the parser to skip such tags instead of
raising a ValidationError. This is done by calling Entrez.read or
Entrez.parse with the argument validate equal to False:
End of explanation
from Bio import Medline
with open("data/pubmed_result1.txt") as handle:
    record = Medline.read(handle)
Explanation: Of course, the information contained in the XML tags that are not in the
DTD are not present in the record returned by Entrez.read.
Specialized parsers
The Bio.Entrez.read() function can parse most (if not all) XML output
returned by Entrez. Entrez typically allows you to retrieve records in
other formats, which may have some advantages compared to the XML format
in terms of readability (or download size).
To request a specific file format from Entrez using
Bio.Entrez.efetch() requires specifying the rettype and/or retmode
optional arguments. The different combinations are described for each
database type on the NCBI efetch
webpage.
One obvious case is you may prefer to download sequences in the FASTA or
GenBank/GenPept plain text formats (which can then be parsed with
Bio.SeqIO, see Sections [sec:SeqIO_GenBank_Online]
and EFetch: Downloading full records from Entrez). For the literature databases, Biopython contains a
parser for the MEDLINE format used in PubMed.
Parsing Medline records {#subsec:entrez-and-medline}
You can find the Medline parser in Bio.Medline. Suppose we want to
parse the file pubmed_result1.txt, containing one Medline record. You
can find this file in Biopython’s Tests\Medline directory. The file
looks like this:
```
PMID- 12230038
OWN - NLM
STAT- MEDLINE
DA  - 20020916
DCOM- 20030606
LR  - 20041117
PUBM- Print
IS  - 1467-5463 (Print)
VI  - 3
IP  - 3
DP  - 2002 Sep
TI  - The Bio* toolkits--a brief overview.
PG  - 296-302
AB  - Bioinformatics research is often difficult to do with commercial software. The
      Open Source BioPerl, BioPython and Biojava projects provide toolkits with
...
```
We first open the file and then parse it:
End of explanation
record["PMID"]
record["AB"]
Explanation: The record now contains the Medline record as a Python dictionary:
End of explanation
help(record)
Explanation: The key names used in a Medline record can be rather obscure; use
End of explanation
from Bio import Medline
with open("data/pubmed_result2.txt") as handle:
    for record in Medline.parse(handle):
        print(record["TI"])
Explanation: for a brief summary.
To parse a file containing multiple Medline records, you can use the
parse function instead:
End of explanation
from Bio import Entrez
Entrez.email = "A.N.Other@example.com"     # Always tell NCBI who you are
handle = Entrez.esearch(db="pubmed", term="biopython")
record = Entrez.read(handle)
record["IdList"]
Explanation: Instead of parsing Medline records stored in files, you can also parse
Medline records downloaded by Bio.Entrez.efetch. For example, let’s
look at all Medline records in PubMed related to Biopython:
End of explanation
idlist = record["IdList"]
handle = Entrez.efetch(db="pubmed", id=idlist, rettype="medline", retmode="text")
Explanation: We now use Bio.Entrez.efetch to download these Medline records:
End of explanation
from Bio import Medline
records = Medline.parse(handle)
for record in records:
    print(record["AU"])
Explanation: Here, we specify rettype="medline", retmode="text" to obtain the
Medline records in plain-text Medline format. Now we use Bio.Medline
to parse these records:
End of explanation
from Bio import Entrez
Entrez.email = "A.N.Other@example.com"     # Always tell NCBI who you are
handle = Entrez.esearch(db="pubmed", term="biopython")
record = Entrez.read(handle)
idlist = record["IdList"]
handle = Entrez.efetch(db="pubmed", id=idlist, rettype="medline", retmode="xml")
records = Entrez.read(handle)
for record in records:
    print(record["MedlineCitation"]["Article"]["ArticleTitle"])
Explanation: For comparison, here we show an example using the XML format:
End of explanation
from Bio import Geo
handle = open("data/GSE16.txt")
records = Geo.parse(handle)
for record in records:
    print(record)
Explanation: Note that in both of these examples, for simplicity we have naively
combined ESearch and EFetch. In this situation, the NCBI would expect
you to use their history feature, as illustrated in
Section History and WebEnv.
Parsing GEO records
GEO (Gene Expression Omnibus) is a
data repository of high-throughput gene expression and hybridization
array data. The Bio.Geo module can be used to parse GEO-formatted
data.
The following code fragment shows how to parse the example GEO file
GSE16.txt into a record and print the record:
End of explanation
from Bio import Entrez
Entrez.email = "A.N.Other@example.com" # Always tell NCBI who you are
handle = Entrez.esearch(db="gds", term="GSE16")
record = Entrez.read(handle)
record["Count"]
record["IdList"]
Explanation: You can search the “gds” database (GEO datasets) with ESearch:
End of explanation
# from Bio import UniGene
# input = open("data/myunigenefile.data")
# record = UniGene.read(input)
Explanation: From the Entrez website, UID “200000016” is GDS16 while the other hit
“100000028” is for the associated platform, GPL28. Unfortunately, at the
time of writing the NCBI don’t seem to support downloading GEO files
using Entrez (not as XML, nor in the Simple Omnibus Format in Text
(SOFT) format).
However, it is actually pretty straight forward to download the GEO
files by FTP or HTTP from http://ftp.ncbi.nih.gov/pub/geo/ instead. In this
case you might want
http://ftp.ncbi.nih.gov/pub/geo/DATA/SOFT/by_series/GSE16/GSE16_family.soft.gz
(a compressed file, see the Python module gzip).
Parsing UniGene records
UniGene is an NCBI database of the transcriptome, with each UniGene
record showing the set of transcripts that are associated with a
particular gene in a specific organism. A typical UniGene record looks
like this:
```
ID          Hs.2
TITLE       N-acetyltransferase 2 (arylamine N-acetyltransferase)
GENE        NAT2
CYTOBAND    8p22
GENE_ID     10
LOCUSLINK   10
HOMOL       YES
EXPRESS      bone| connective tissue| intestine| liver| liver tumor| normal| soft tissue/muscle tissue tumor| adult
RESTR_EXPR   adult
CHROMOSOME  8
STS         ACC=PMC310725P3 UNISTS=272646
STS         ACC=WIAF-2120 UNISTS=44576
STS         ACC=G59899 UNISTS=137181
...
STS         ACC=GDB:187676 UNISTS=155563
PROTSIM     ORG=10090; PROTGI=6754794; PROTID=NP_035004.1; PCT=76.55; ALN=288
PROTSIM     ORG=9796; PROTGI=149742490; PROTID=XP_001487907.1; PCT=79.66; ALN=288
PROTSIM     ORG=9986; PROTGI=126722851; PROTID=NP_001075655.1; PCT=76.90; ALN=288
...
PROTSIM     ORG=9598; PROTGI=114619004; PROTID=XP_519631.2; PCT=98.28; ALN=288
SCOUNT      38
SEQUENCE    ACC=BC067218.1; NID=g45501306; PID=g45501307; SEQTYPE=mRNA
SEQUENCE    ACC=NM_000015.2; NID=g116295259; PID=g116295260; SEQTYPE=mRNA
SEQUENCE    ACC=D90042.1; NID=g219415; PID=g219416; SEQTYPE=mRNA
SEQUENCE    ACC=D90040.1; NID=g219411; PID=g219412; SEQTYPE=mRNA
SEQUENCE    ACC=BC015878.1; NID=g16198419; PID=g16198420; SEQTYPE=mRNA
SEQUENCE    ACC=CR407631.1; NID=g47115198; PID=g47115199; SEQTYPE=mRNA
SEQUENCE    ACC=BG569293.1; NID=g13576946; CLONE=IMAGE:4722596; END=5'; LID=6989; SEQTYPE=EST; TRACE=44157214
...
SEQUENCE    ACC=AU099534.1; NID=g13550663; CLONE=HSI08034; END=5'; LID=8800; SEQTYPE=EST
//
```
This particular record shows the set of transcripts (shown in the
SEQUENCE lines) that originate from the human gene NAT2, encoding en
N-acetyltransferase. The PROTSIM lines show proteins with significant
similarity to NAT2, whereas the STS lines show the corresponding
sequence-tagged sites in the genome.
To parse UniGene files, use the Bio.UniGene module:
<font color='red'>TODO: Need a working example</font>
End of explanation
# record.ID
# record.title
Explanation: The record returned by UniGene.read is a Python object with
attributes corresponding to the fields in the UniGene record. For
example,
End of explanation
# record.sts[0].acc
# record.sts[0].unists
Explanation: The EXPRESS and RESTR_EXPR lines are stored as Python lists of
strings:
```
['bone', 'connective tissue', 'intestine', 'liver', 'liver tumor', 'normal', 'soft tissue/muscle tissue tumor', 'adult']
```
Specialized objects are returned for the STS, PROTSIM, and
SEQUENCE lines, storing the keys shown in each line as attributes:
End of explanation
# from Bio import UniGene
# input = open("unigenerecords.data")
# records = UniGene.parse(input)
# for record in records:
#     print(record.ID)
Explanation: and similarly for the PROTSIM and SEQUENCE lines.
To parse a file containing more than one UniGene record, use the parse
function in Bio.UniGene:
<font color='red'>TODO: Need a working example</font>
End of explanation
from Bio import Entrez
Entrez.email = "A.N.Other@example.com"     # Always tell NCBI who you are
handle = Entrez.egquery(term="orchid")
record = Entrez.read(handle)
for row in record["eGQueryResult"]:
    if row["DbName"]=="pubmed":
        print(row["Count"])
Explanation: Using a proxy
Normally you won’t have to worry about using a proxy, but if this is an
issue on your network here is how to deal with it. Internally,
Bio.Entrez uses the standard Python library urllib for accessing the
NCBI servers. This will check an environment variable called
http_proxy to configure any simple proxy automatically. Unfortunately
this module does not support the use of proxies which require
authentication.
You may choose to set the http_proxy environment variable once (how
you do this will depend on your operating system). Alternatively you can
set this within Python at the start of your script, for example:
```
import os
os.environ["http_proxy"] = "http://proxyhost.example.com:8080"
```
See the urllib
documentation for
more details.
Examples
PubMed and Medline {#subsec:pub_med}
If you are in the medical field or interested in human issues (and many
times even if you are not!), PubMed
(http://www.ncbi.nlm.nih.gov/PubMed/) is an excellent source of all
kinds of goodies. So like other things, we’d like to be able to grab
information from it and use it in Python scripts.
In this example, we will query PubMed for all articles having to do with
orchids (see section [sec:orchids] for our motivation). We first check
how many of such articles there are:
End of explanation
handle = Entrez.esearch(db="pubmed", term="orchid", retmax=463)
record = Entrez.read(handle)
idlist = record["IdList"]
print("The first 10 Id's containing all of the PubMed IDs of articles related to orchids:\n {}".format(idlist[:10]))
Explanation: Now we use the Bio.Entrez.efetch function to download the PubMed IDs
of these 463 articles:
End of explanation
from Bio import Medline
handle = Entrez.efetch(db="pubmed", id=idlist, rettype="medline")
records = Medline.parse(handle)
Explanation: Now that we’ve got them, we obviously want to get the corresponding
Medline records and extract the information from them. Here, we’ll
download the Medline records in the Medline flat-file format, and use
the Bio.Medline module to parse them:
End of explanation
records = list(records)
Explanation: NOTE - We’ve just done a separate search and fetch here, the NCBI much
prefer you to take advantage of their history support in this situation.
See Section History and WebEnv.
Keep in mind that records is an iterator, so you can iterate through
the records only once. If you want to save the records, you can convert
them to a list:
End of explanation
for record in records:
    print("title:", record.get("TI", "?"))
    print("authors:", record.get("AU", "?"))
    print("source:", record.get("SO", "?"))
    print("")
Explanation: Let’s now iterate over the records to print out some information about
each record:
End of explanation
search_author = "Waits T"
for record in records:
    if not "AU" in record:
        continue
    if search_author in record["AU"]:
        print("Author %s found: %s" % (search_author, record["SO"]))
Explanation: The output for this looks like:
```
title: Sex pheromone mimicry in the early spider orchid (ophrys sphegodes):
patterns of hydrocarbons as the key mechanism for pollination by sexual
deception [In Process Citation]
authors: ['Schiestl FP', 'Ayasse M', 'Paulus HF', 'Lofstedt C', 'Hansson BS',
'Ibarra F', 'Francke W']
source: J Comp Physiol [A] 2000 Jun;186(6):567-74
```
Especially interesting to note is the list of authors, which is returned
as a standard Python list. This makes it easy to manipulate and search
using standard Python tools. For instance, we could loop through a whole
bunch of entries searching for a particular author with code like the
following:
End of explanation
from Bio import Entrez
Entrez.email = "A.N.Other@example.com"     # Always tell NCBI who you are
handle = Entrez.egquery(term="Cypripedioideae")
record = Entrez.read(handle)
for row in record["eGQueryResult"]:
    if row["DbName"]=="nuccore":
        print(row["Count"])
Explanation: Hopefully this section gave you an idea of the power and flexibility of
the Entrez and Medline interfaces and how they can be used together.
Searching, downloading, and parsing Entrez Nucleotide records {#subsec:entrez_example_genbank}
Here we’ll show a simple example of performing a remote Entrez query. In
section [sec:orchids] of the parsing examples, we talked about using
NCBI’s Entrez website to search the NCBI nucleotide databases for info
on Cypripedioideae, our friends the lady slipper orchids. Now, we’ll
look at how to automate that process using a Python script. In this
example, we’ll just show how to connect, get the results, and parse
them, with the Entrez module doing all of the work.
First, we use EGQuery to find out the number of results we will get
before actually downloading them. EGQuery will tell us how many search
results were found in each of the databases, but for this example we are
only interested in nucleotides:
End of explanation
from Bio import Entrez
handle = Entrez.esearch(db="nucleotide", term="Cypripedioideae", retmax=814)
record = Entrez.read(handle)
Explanation: So, we expect to find 814 Entrez Nucleotide records (this is the number
I obtained in 2008; it is likely to increase in the future). If you find
some ridiculously high number of hits, you may want to reconsider if you
really want to download all of them, which is our next step:
End of explanation
print(record.keys())
Explanation: Here, record is a Python dictionary containing the search results and
some auxiliary information. Just for information, let’s look at what is
stored in this dictionary:
End of explanation
print(record["Count"])
Explanation: First, let’s check how many results were found:
End of explanation
len(record["IdList"])
Explanation: which is the number we expected. The 814 results are stored in
record['IdList']:
End of explanation
record["IdList"][:5]
Explanation: Let’s look at the first five results:
End of explanation
idlist = ",".join(record["IdList"][:5])
print(idlist)
handle = Entrez.efetch(db="nucleotide", id=idlist, retmode="xml")
records = Entrez.read(handle)
len(records)
Explanation: [sec:entrez-batched-efetch] We can download these records using
efetch. While you could download these records one by one, to reduce
the load on NCBI’s servers, it is better to fetch a bunch of records at
the same time, shown below. However, in this situation you should
ideally be using the history feature described later in
Section History and WebEnv.
End of explanation
print(records[0].keys())
print(records[0]["GBSeq_primary-accession"])
print(records[0]["GBSeq_other-seqids"])
print(records[0]["GBSeq_definition"])
print(records[0]["GBSeq_organism"])
Explanation: Each of these records corresponds to one GenBank record.
End of explanation
from Bio import Entrez
Entrez.email = "A.N.Other@example.com"     # Always tell NCBI who you are
handle = Entrez.egquery(term="Opuntia AND rpl16")
record = Entrez.read(handle)
for row in record["eGQueryResult"]:
    if row["DbName"]=="nuccore":
        print(row["Count"])
Explanation: You could use this to quickly set up searches – but for heavy usage, see
Section History and WebEnv.
Searching, downloading, and parsing GenBank records {#sec:entrez-search-fetch-genbank}
The GenBank record format is a very popular method of holding
information about sequences, sequence features, and other associated
sequence information. The format is a good way to get information from
the NCBI databases at http://www.ncbi.nlm.nih.gov/.
In this example we’ll show how to query the NCBI databases,to retrieve
the records from the query, and then parse them using Bio.SeqIO -
something touched on in Section [sec:SeqIO_GenBank_Online]. For
simplicity, this example does not take advantage of the WebEnv history
feature – see Section History and WebEnv for this.
First, we want to make a query and find out the ids of the records to
retrieve. Here we’ll do a quick search for one of our favorite
organisms, Opuntia (prickly-pear cacti). We can do quick search and
get back the GIs (GenBank identifiers) for all of the corresponding
records. First we check how many records there are:
End of explanation
handle = Entrez.esearch(db="nuccore", term="Opuntia AND rpl16")
record = Entrez.read(handle)
gi_list = record["IdList"]
gi_list
Explanation: Now we download the list of GenBank identifiers:
End of explanation
gi_str = ",".join(gi_list)
handle = Entrez.efetch(db="nuccore", id=gi_str, rettype="gb", retmode="text")
Explanation: Now we use these GIs to download the GenBank records - note that with
older versions of Biopython you had to supply a comma separated list of
GI numbers to Entrez, as of Biopython 1.59 you can pass a list and this
is converted for you:
End of explanation
text = handle.read()
print(text)
Explanation: If you want to look at the raw GenBank files, you can read from this
handle and print out the result:
End of explanation
from Bio import SeqIO
handle = Entrez.efetch(db="nuccore", id=gi_str, rettype="gb", retmode="text")
records = SeqIO.parse(handle, "gb")
Explanation: In this case, we are just getting the raw records. To get the records in
a more Python-friendly form, we can use Bio.SeqIO to parse the GenBank
data into SeqRecord objects, including SeqFeature objects (see
Chapter [chapter:Bio.SeqIO]):
End of explanation
for record in records:
    print("%s, length %i, with %i features" \
    % (record.name, len(record), len(record.features)))
Explanation: We can now step through the records and look at the information we are
interested in:
End of explanation
from Bio import Entrez
Entrez.email = "A.N.Other@example.com"     # Always tell NCBI who you are
handle = Entrez.esearch(db="Taxonomy", term="Cypripedioideae")
record = Entrez.read(handle)
record["IdList"]
record["IdList"][0]
Explanation: Using these automated query retrieval functionality is a big plus over
doing things by hand. Although the module should obey the NCBI’s max
three queries per second rule, the NCBI have other recommendations like
avoiding peak hours. See Section [sec:entrez-guidelines]. In
particular, please note that for simplicity, this example does not use
the WebEnv history feature. You should use this for any non-trivial
search and download work, see Section History and WebEnv.
Finally, if plan to repeat your analysis, rather than downloading the
files from the NCBI and parsing them immediately (as shown in this
example), you should just download the records once and save them to
your hard disk, and then parse the local file.
Finding the lineage of an organism
Staying with a plant example, let’s now find the lineage of the
Cypripedioideae orchid family. First, we search the Taxonomy database
for Cypripedioideae, which yields exactly one NCBI taxonomy identifier:
End of explanation
handle = Entrez.efetch(db="Taxonomy", id="158330", retmode="xml")
records = Entrez.read(handle)
Explanation: Now, we use efetch to download this entry in the Taxonomy database,
and then parse it:
End of explanation
records[0].keys()
Explanation: Again, this record stores lots of information:
End of explanation
records[0]["Lineage"]
Explanation: We can get the lineage directly from this record:
End of explanation
from Bio import Entrez
Entrez.email = "history.user@example.com"
search_handle = Entrez.esearch(db="nucleotide",term="Opuntia[orgn] and rpl16", usehistory="y")
search_results = Entrez.read(search_handle)
search_handle.close()
Explanation: The record data contains much more than just the information shown here
- for example look under LineageEx instead of Lineage and you’ll get
the NCBI taxon identifiers of the lineage entries too.
Using the history and WebEnv
Often you will want to make a series of linked queries. Most typically,
running a search, perhaps refining the search, and then retrieving
detailed search results. You can do this by making a series of
separate calls to Entrez. However, the NCBI prefer you to take advantage
of their history support - for example combining ESearch and EFetch.
Another typical use of the history support would be to combine EPost and
EFetch. You use EPost to upload a list of identifiers, which starts a
new history session. You then download the records with EFetch by
referring to the session (instead of the identifiers).
Searching for and downloading sequences using the history
Suppose we want to search and download all the Opuntia rpl16
nucleotide sequences, and store them in a FASTA file. As shown in
Section [sec:entrez-search-fetch-genbank], we can naively combine
Bio.Entrez.esearch() to get a list of GI numbers, and then call
Bio.Entrez.efetch() to download them all.
However, the approved approach is to run the search with the history
feature. Then, we can fetch the results by reference to the search
results - which the NCBI can anticipate and cache.
To do this, call Bio.Entrez.esearch() as normal, but with the
additional argument of usehistory="y",
End of explanation
gi_list = search_results["IdList"]
count = int(search_results["Count"])
assert count == len(gi_list)
print("The WebEnv is {}".format(search_results["WebEnv"]))
print("The QueryKey is {}".format(search_results["QueryKey"]))
Explanation: When you get the XML output back, it will still include the usual search
results. However, you also get given two additional pieces of information, the
WebEnv session cookie, and the QueryKey:
End of explanation
from Bio import Entrez
Entrez.email = "A.N.Other@example.com"
pmid = "14630660"
results = Entrez.read(Entrez.elink(dbfrom="pubmed", db="pmc",
LinkName="pubmed_pmc_refs", from_uid=pmid))
pmc_ids = [link["Id"] for link in results[0]["LinkSetDb"][0]["Link"]]
pmc_ids
Explanation: Having stored these values in variables <span>session_cookie</span> and
<span>query_key</span> we can use them as parameters to
Bio.Entrez.efetch() instead of giving the GI numbers as identifiers.
While for small searches you might be OK downloading everything at once,
it is better to download in batches. You use the <span>retstart</span>
and <span>retmax</span> parameters to specify which range of search
results you want returned (starting entry using zero-based counting, and
maximum number of results to return). Sometimes you will get
intermittent errors from Entrez, HTTPError 5XX, we use a try except
pause retry block to address this. For example,
```
from Bio import Entrez
import time
try:
    from urllib.error import HTTPError  # for Python 3
except ImportError:
    from urllib2 import HTTPError  # for Python 2
batch_size = 3
out_handle = open("orchid_rpl16.fasta", "w")
for start in range(0, count, batch_size):
    end = min(count, start+batch_size)
    print("Going to download record %i to %i" % (start+1, end))
    attempt = 1
    while attempt <= 3:
        try:
            fetch_handle = Entrez.efetch(db="nucleotide", rettype="fasta", retmode="text",
                                         retstart=start, retmax=batch_size,
                                         webenv=webenv, query_key=query_key)
        except HTTPError as err:
            if 500 <= err.code <= 599:
                print("Received error from server %s" % err)
                print("Attempt %i of 3" % attempt)
                attempt += 1
                time.sleep(15)
            else:
                raise
    data = fetch_handle.read()
    fetch_handle.close()
    out_handle.write(data)
out_handle.close()
```
For illustrative purposes, this example downloaded the FASTA records in
batches of three. Unless you are downloading genomes or chromosomes, you
would normally pick a larger batch size.
Searching for and downloading abstracts using the history
Here is another history example, searching for papers published in the
last year about the Opuntia, and then downloading them into a file in
MedLine format:
```
from Bio import Entrez
import time
try:
    from urllib.error import HTTPError  # for Python 3
except ImportError:
    from urllib2 import HTTPError  # for Python 2
Entrez.email = "history.user@example.com"
search_results = Entrez.read(Entrez.esearch(db="pubmed",
                                            term="Opuntia[ORGN]",
                                            reldate=365, datetype="pdat",
                                            usehistory="y"))
count = int(search_results["Count"])
print("Found %i results" % count)
batch_size = 10
out_handle = open("recent_orchid_papers.txt", "w")
for start in range(0,count,batch_size):
    end = min(count, start+batch_size)
    print("Going to download record %i to %i" % (start+1, end))
    attempt = 1
    while attempt <= 3:
        try:
            fetch_handle = Entrez.efetch(db="pubmed",rettype="medline",
                                         retmode="text",retstart=start,
                                         retmax=batch_size,
                                         webenv=search_results["WebEnv"],
                                         query_key=search_results["QueryKey"])
        except HTTPError as err:
            if 500 <= err.code <= 599:
                print("Received error from server %s" % err)
                print("Attempt %i of 3" % attempt)
                attempt += 1
                time.sleep(15)
            else:
                raise
    data = fetch_handle.read()
    fetch_handle.close()
    out_handle.write(data)
out_handle.close()
```
At the time of writing, this gave 28 matches - but because this is a
date dependent search, this will of course vary. As described in
Section [subsec:entrez-and-medline] above, you can then use
Bio.Medline to parse the saved records.
Searching for citations {#sec:elink-citations}
Back in Section [sec:elink] we mentioned ELink can be used to search
for citations of a given paper. Unfortunately this only covers journals
indexed for PubMed Central (doing it for all the journals in PubMed
would mean a lot more work for the NIH). Let’s try this for the
Biopython PDB parser paper, PubMed ID 14630660:
End of explanation
results2 = Entrez.read(Entrez.elink(dbfrom="pmc", db="pubmed", LinkName="pmc_pubmed",
from_uid=",".join(pmc_ids)))
pubmed_ids = [link["Id"] for link in results2[0]["LinkSetDb"][0]["Link"]]
pubmed_ids
Explanation: Great - eleven articles. But why hasn’t the Biopython application note
been found (PubMed ID 19304878)? Well, as you might have guessed from
the variable names, there are not actually PubMed IDs, but PubMed
Central IDs. Our application note is the third citing paper in that
list, PMCID 2682512.
So, what if (like me) you’d rather get back a list of PubMed IDs? Well
we can call ELink again to translate them. This becomes a two step
process, so by now you should expect to use the history feature to
accomplish it (Section History and WebEnv).
But first, taking the more straightforward approach of making a second
(separate) call to ELink:
End of explanation
<END_TASK>