How to scrape images from a aspx page?
<p>i am trying to <strong>scrape</strong> images from a aspx page i have this code that scrapes images from normal webpage but can't scrape aspx page cause i need to send http post requests to the aspx page i can't figure out how to do that even after reading few threads this is the original code</p>
<pre><code>from bs4 import BeautifulSoup as bs
import urlparse
import urllib2
from urllib import urlretrieve
import os
import sys
import subprocess
import re


def thefunc(url, out_folder):

    c = False
</code></pre>
<p><strong>i have already defined headers for aspx page and a if statement that distinguished between normal page and aspx page</strong></p>
<pre><code>    select =  raw_input('Is this a .net  aspx page ? y/n : ')
    if select.lower().startswith('y'):
        usin = raw_input('Specify origin of .net page : ')
        usaspx = raw_input('Specify aspx page url : ')
</code></pre>
<p><strong>the header for aspx page</strong></p>
<pre><code>        headdic = {
            'Accept':'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8',
            'Origin': usin,
            'User-Agent': 'Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.17 (KHTML, like Gecko)  Chrome/24.0.1312.57 Safari/537.17',
            'Content-Type': 'application/x-www-form-urlencoded',
            'Referer': usaspx,
            'Accept-Encoding': 'gzip,deflate,sdch',
            'Accept-Language': 'en-US,en;q=0.8',
            'Accept-Charset': 'ISO-8859-1,utf-8;q=0.7,*;q=0.3'
        }
        c = True

    if c:
        req = urllib2.Request(url, headers=headic)
    else:
        req = urllib2.Request(url, headers={'User-Agent' : &quot;Magic Browser&quot;})
    resp = urllib2.urlopen(req)
    
    soup = bs(resp, 'lxml')
    
    parsed = list(urlparse.urlparse(url))

    print '\n',len(soup.findAll('img')), 'images are about to be downloaded'

    for image in soup.findAll(&quot;img&quot;):
        
        print &quot;Image: %(src)s&quot; % image
        
        filename = image[&quot;src&quot;].split(&quot;/&quot;)[-1]
        
        parsed[2] = image[&quot;src&quot;]
        
        outpath = os.path.join(out_folder, filename)

        try:
        
            if image[&quot;src&quot;].lower().startswith(&quot;http&quot;):
                urlretrieve(image[&quot;src&quot;], outpath)
            else:
                urlretrieve(urlparse.urlunparse(parsed), outpath)
        except:
            print 'OOPS missed one for some reason !!'
            pass


try:
    put =  raw_input('Please enter the page url : ')
    reg1 = re.compile('^http*',re.IGNORECASE)
    reg1.match(put)
except:
    print('Type the url carefully !!')
    sys.exit()
fol = raw_input('Enter the foldername to save the images : ')
if os.path.isdir(fol):
    thefunc(put, fol)
else:
    subprocess.call('mkdir', fol)
    thefunc(put, fol)
</code></pre>
<p><em><strong>i have made few modifications for aspx detection and creating the header for the aspx page but how to modify next i am stuck here</strong></em></p>
<p><code>***here is the aspx page link***</code> <a href="http://www.foxrun.com.au/Products/Cylinders_with_Gadgets.aspx" rel="nofollow noreferrer">http://www.foxrun.com.au/Products/Cylinders_with_Gadgets.aspx</a></p>
<p><em><strong>sorry if i am not clear as you can see i am new to programming, the question i am asking is how can i get the images i get from the aspx page when i am clicking the next page button in the browser cause if i can only scrape one page cause the url is not changing unless i send a http post somehow to tell the page to show the next page with new pictures, because the url stays the same i hope i am clear</strong></em></p>