How to extract text from a webpage using python 2.7?
<p>I'm trying to programmatically extract text from this webpage which describes a genome assembly in the public archive: </p>

<p><a href="http://www.ebi.ac.uk/ena/data/view/ERS019623" rel="nofollow noreferrer">http://www.ebi.ac.uk/ena/data/view/ERS019623</a></p>

<p>I have thousands of assemblies that I want to track down and extract the study accession, which is the code on the far left of the table beginning with "PRJ". The URL for each of these assemblies is of the same format as the one above, i.e. "<a href="http://www.ebi.ac.uk/ena/data/view/ERS" rel="nofollow noreferrer">http://www.ebi.ac.uk/ena/data/view/ERS</a>******". I have the ERS code for each of my assemblies so I can construct the URL for each one. </p>

<p>I've tried a few different methods, firstly if you add "&amp;display=XML" to the end of the URL it prints the XML (or at least I'm presuming that it's printing the XML for the entire page, because the problem is that the study accession "PRJ******" is no where to be seen here). I had utilised this to extract another code that I needed from the same webpage, the run accession which is always of the format "ERR******" using the below code:   </p>

<pre><code>import urllib2
from bs4 import BeautifulSoup
import re
import csv

with open('/Users/bj5/Desktop/web_scrape_test.csv','rb') as f:
reader = csv.reader(f) #opens csv containig list of ERS numbers
for row in reader:
    sample = row[0] #reads index 0 (1st row)
    ERSpage = "http://www.ebi.ac.uk/ena/data/view/" + sample + "&amp;display=xml" #creates URL using ERS number from 1st row
    page = urllib2.urlopen(ERSpage) #opens url and assigns it to variable page
    soup = BeautifulSoup(page, "html.parser") #parses the html/xml from page and assigns it to variable called soup
    page_text = soup.text #returns text from variable soup, i.e. no tags
    ERS = re.search('ERS......', page_text, flags=0).group(0) #returns first ERS followed by six wildcards
    ERR = re.search('ERR......', page_text, flags=0).group(0) #retursn first ERR followed by six wildcards
    print ERS + ',' + ERR + ',' + "http://www.ebi.ac.uk/ena/data/view/" + sample #prints ERS,ERR,URL
</code></pre>

<p>This worked very well, but as the study accession is not in the XML I can't use it to access this. </p>

<p>I also attempted to use BeautifulSoup again to download the HTML by doing this: </p>

<pre><code>from bs4 import BeautifulSoup
from urllib2 import urlopen

BASE_URL = "http://www.ebi.ac.uk/ena/data/view/ERS019623"

def get_category_links(section_url):
    html = urlopen(section_url).read()
    soup = BeautifulSoup(html, "lxml")
    print soup

get_category_links(BASE_URL)
</code></pre>

<p>But again I can't see the study accession in the output from this either...</p>

<p>I have also attempted to use a different python module, lxml, to parse the XML and HTML but haven't had any luck there either. </p>

<p>When I right click and inspect element on the page I can find the study accession by doing ctrl+F -> PRJ. </p>

<p>So my question is this: what is the code that I'm looking at in inspect element, XML or HTML (or something else)? Why does it look different to the code that prints in my console when I try and use BeautifulSoup to parse HTML? And finally how can I scrape the study accessions (PRJ******) from these webpages?</p>

<p>(I've only been coding for a couple of months and I'm entirely self-taught so apologies for the slightly confused nature of this question but I hope I've got across what it is that I'm trying to do. Any suggestions or advice would be much appreciated.)  </p>