How to Retry 503 response using Scrapy DownloadMiddleware?
<p>In my I have this</p>

<pre><code>from scrapy.downloadermiddlewares.retry import RetryMiddleware

class Retry(RetryMiddleware):

    def process_response(self, request, response, spider):

        if response.status == '503':
            logger.error("503 status returned: " + response.url)
            return self._retry(request,response, spider) or response

        logger.debug("response.status = "+str(response.status)+" from URL "+str(response.url))
        logger.debug(response.headers)

        return super(Retry, self).process_response(request, response, spider)


    def _retry(self, request,response, spider):

        logger.debug("Deleting session "+str(request.meta['sessionId']))
        self.delete_session(request.meta['sessionId'])

        logger.debug("Retrying URL: %(request)s", {'request': request})
        logger.debug("Response headers were:")
        logger.debug(request.headers)

        retryreq = request.copy()
        retryreq.headers['Authorization'] = crawlera_auth.strip()
        retryreq.headers['X-Crawlera-Session'] = 'create'

        retryreq.dont_filter = True
        return retryreq
</code></pre>

<p>And in my <code>settings.py</code> I have this</p>

<pre><code>DOWNLOADER_MIDDLEWARES = {
    'craigslist_tickets.retrymiddleware.Retry': 100,
    'craigslist_tickets.crawlera_proxy_middleware.CrawleraProxyMiddleware': 200
}
</code></pre>

<p>I can see output like <code>response.status =  200</code> for all URLs which are scraped successfully, but for the URLs which return <code>500</code> don't even are passing through <code>process_response</code> </p>

<p>I can only see in terminal </p>

<p><code>[scrapy] DEBUG: Retrying &lt;GET http:website.com&gt; (failed 1 times): 503 Service Unavailable</code></p>

<p><strong>SHORT QUESTION:</strong></p>

<p>I want to scrape URLs which return <code>503</code> again by passing through <code>process_response</code> method of my custom class <code>Retry</code></p>