How to scrape for specific HTML span elements that get loaded with JQuery
<p>I am writing a python app that scrapes sensor data from a web-app that I made. I want to use lxml to gather certain span elements from the HTML that get updated periodically with a JQuery script. It seems as though lxml does not grab the text that gets updated with JQuery because I only retrieve the empty string that the original HTML is loaded with. The values from JQuery are reflected when viewing the webpage, but not on the web scraping tool.  How can I consistently retrieve the data that is being updated with JQuery?</p>

<p>HTML</p>

<pre><code>            &lt;!--Blower Speed Well --&gt;
            &lt;div class = "col-xs-4"&gt;
                &lt;div class = "well" id="bspeed_well"&gt;
                    &lt;span class = "tex" id="bspeed_text"&gt;Blower Speed: &lt;/span&gt;
                    &lt;span class = "val" id="bspeed_data"&gt;&lt;/span&gt;
                &lt;/div&gt;
            &lt;/div&gt;
</code></pre>

<p>JQuery to update HTML element</p>

<pre><code>//Gather Data from main.xml and store into variables
$.ajax({
    type: "GET",
    url: "static/main.xml",
    //dataType: "xml",
    async: false,
    cache: false,
    success: function(xml){
        $(xml).find('item').each(function(){
            var id = $(this).find('id').text();
            switch(id){
                   case "blower_speed":
                    bspeed = $(this).find('value').text();
                    break;
        }
});
$("#bspeed_data").text(bspeed);
</code></pre>

<p>Python lxml query </p>

<pre><code>page = requests.get('http://192.168.1.72:5000')
tree = html.fromstring(page.content)
blower_speed = tree.xpath('//span[@id="bspeed_data"]/text()')
print "Blower Speed: ", blower_speed
</code></pre>

<p>Result</p>

<blockquote>
  <p>Blower Speed:  []</p>
</blockquote>