Python BeautifulSoup - Scraping Google Finance historical data
<p>I was trying to scrap Google Finance historical data. I was need of to total number of rows, which is located along with the pagination. The following is the div tag which is responsible for displaying the total number of rows:</p>

<pre><code>&lt;div class="tpsd"&gt;1 - 30 of 1634 rows&lt;/div&gt;
</code></pre>

<p>I tried using the following code to get the data, but its returning an empty list:</p>

<pre><code>soup.find_all('div', 'tpsd')
</code></pre>

<p>I tried getting the entire table but even then I was not successful, when I checked the page source I was able to find the value inside a JavaScript function. When I Googled how to get values from script tag, it was mentioned to used regex. So, I tried using regex and the following is my code:</p>

<pre><code>import requests
import re
from bs4 import BeautifulSoup
r = requests.get('https://www.google.com/finance/historical?cid=13564339&amp;startdate=Jan+01%2C+2010&amp;enddate=Aug+18%2C+2016&amp;num=30&amp;ei=ilC1V6HlPIasuASP9Y7gAQ')
soup = BeautifulSoup(r.content,'lxml')
var = soup.find_all("script")[8].string
a = re.compile('google.finance.applyPagination\((.*)\'http', re.DOTALL)
b =  a.search(var)
num = b.group(1)
print(num.replace(',','').split('\n')[3])
</code></pre>

<p>I am able to get the values which I want, but my doubt is whether the above code which I used to get the values is correct, or is there any other way better way. Kindly help.</p>