Persist Session Cookie in Scrapy
<p>I am scraping a site that has an accept terms form that I need to click through. When I click the button I am redirected to the resource that needs to be scraped. I have the basic mechanics working, that is the initial click through works and I get a session and all goes well until the session times out. Then for some reason Scrapy does get redirected but the response URL doesn't get updated so I get duplicate items since I am using the URL to check for duplication. </p>

<p>For example the URL I am requesting is:</p>

<pre><code>https://some-internal-web-page/Records/Details/119ce2b7-35b4-4c63-8bd2-2bfbf77299a8
</code></pre>

<p>But when the session expires I get:</p>

<pre><code>https://some-internal-web-page/?returnUrl=%2FRecords%2FDetails%2F119ce2b7-35b4-4c63-8bd2-2bfbf77299a8
</code></pre>

<p>Here is my code:</p>

<pre><code> # function to get through accept dialog
def parse(self, response):
    yield FormRequest.from_response(response, formdata={"value":"Accept"}, callback=self.after_accept)

# function to parse markup
def after_accept(self, response):

    global latest_inspection_date

    urls = ['http://some-internal-web-page/Records?SearchText=&amp;SortMode=MostRecentlyHired&amp;page=%s&amp;PageSize=25' % page for page in xrange(1,500)]

    for u in urls:
        yield Request( u, callback=self.parse_list )
</code></pre>

<p>So my question is, how do I persist and/or refresh the session cookie so that I don't get the redirect URL instead of the URL I need.</p>