Pause URL request Downloads
<pre><code>import urllib.request
import re
import csv
import pandas as pd
from bs4 import BeautifulSoup

columns = []
data = []
f = open('companylist.csv')
csv_f = csv.reader(f)

for row in csv_f:
    stocklist = row
    print(stocklist)

    for s in stocklist:
        print('http://finance.yahoo.com/q?s='+s)
        optionsUrl = urllib.request.urlopen('http://finance.yahoo.com/q?s='+s).read()
        soup = BeautifulSoup(optionsUrl, "html.parser")
        stocksymbol = ['Symbol:', s]
        optionsTable = [stocksymbol]+[
        [x.text for x in y.parent.contents]
        for y in soup.findAll('td', attrs={'class': 'yfnc_tabledata1','rtq_table': ''})
        ]
        if not columns:
            columns = [o[0] for o in optionsTable] #list(my_df.loc[0])
        data.append(o[1] for o in optionsTable)




# create DataFrame from data
df = pd.DataFrame(data, columns=columns)
df.to_csv('test.csv', index=False)
</code></pre>

<p>The scripts works fine when I have about 200 to 300 stocks, but my company list has around 6000 symbols.</p>

<ol>
<li>Is there a way I can download chunks of data, say like 200 stocks at a time, pause for while, and then resume the download again?</li>
<li>The export is one stock at a time; how do I write 200 at a time, and append the next batch to the initial batch (for the CSV)?</li>
</ol>