Scrapy - Can't parse URL after login
<p>I have a problem with a Scrapy script. I successfully log to the website I need to crawl, but after login I stay blocked on the start_urls/login_page instead of parsing the URL "<a href="http://www.tm-alumni.eu/#/annuaire/diplomes?user_type=1&amp;filterGeo=1&amp;activation_status=-1&amp;view=tromb&amp;page=1" rel="nofollow">http://www.tm-alumni.eu/#/annuaire/diplomes?user_type=1&amp;filterGeo=1&amp;activation_status=-1&amp;view=tromb&amp;page=1</a>"</p>

<p>I saw a lot of topics about this on Stackoverflow but none of them had this same problem...</p>

<p>Here is the code I'm using:</p>

<pre><code>import re
from scrapy.spiders.init import InitSpider
from scrapy.http import Request, FormRequest
from items import temaSpiderItem

class temaSpider(InitSpider):
    name = 'temaSpider'
    allowed_domains = ['http://tm-alumni.eu', 'http://www.tm-alumni.eu/#/annuaire/' ]
    start_urls = ['http://www.tm-alumni.eu/']
    login_page = 'http://www.tm-alumni.eu/'
    directory_page = 'http://www.tm-alumni.eu/#/annuaire/diplomes?user_type=1&amp;filterGeo=1&amp;activation_status=-1&amp;view=tromb&amp;page=1'

    def init_request(self):
        return Request(url=self.login_page, callback=self.login)

    def login(self, response):
        return FormRequest.from_response(response,
                    formxpath="//form[@id='loginform']",
                    formdata={'username': 'email', 'password': 'password'},
                    callback=self.check_login,
                    dont_filter=True)

    def check_login(self, response):
        if "My Name" in response.body:
            self.log("=========Successfully logged in.=========")
            return Request(url=self.directory_page, callback=self.parse_directory, dont_filter=True)
        else:
            self.log("=========An error in login occurred.=========")

    def parse_directory(self, response):
        self.log("=========Data is flowing.=========")
        self.log(response.url)
</code></pre>

<p>And here is what I find in the console:</p>

<pre><code>2016-01-05 11:48:49 [scrapy] INFO: Scrapy 1.0.3 started (bot: scrapybot)
2016-01-05 11:48:49 [scrapy] INFO: Optional features available: ssl, http11, boto
2016-01-05 11:48:49 [scrapy] INFO: Overridden settings: {'FEED_FORMAT': 'json', 'FEED_URI': 'Alumnis.json'}
2016-01-05 11:48:50 [scrapy] INFO: Enabled extensions: CloseSpider, FeedExporter, TelnetConsole, LogStats, CoreStats, SpiderState
2016-01-05 11:48:50 [boto] DEBUG: Retrieving credentials from metadata server.
2016-01-05 11:48:51 [boto] ERROR: Caught exception reading instance data
Traceback (most recent call last):
  File "C:\Users\nmitchell\AppData\Local\Continuum\Anaconda\lib\site-packages\boto\utils.py", line 210, in retry_url
    r = opener.open(req, timeout=timeout)
  File "C:\Users\nmitchell\AppData\Local\Continuum\Anaconda\lib\urllib2.py", line 431, in open
    response = self._open(req, data)
  File "C:\Users\nmitchell\AppData\Local\Continuum\Anaconda\lib\urllib2.py", line 449, in _open
    '_open', req)
  File "C:\Users\nmitchell\AppData\Local\Continuum\Anaconda\lib\urllib2.py", line 409, in _call_chain
    result = func(*args)
  File "C:\Users\nmitchell\AppData\Local\Continuum\Anaconda\lib\urllib2.py", line 1227, in http_open
    return self.do_open(httplib.HTTPConnection, req)
  File "C:\Users\nmitchell\AppData\Local\Continuum\Anaconda\lib\urllib2.py", line 1197, in do_open
    raise URLError(err)
URLError: &lt;urlopen error timed out&gt;
2016-01-05 11:48:51 [boto] ERROR: Unable to read instance data, giving up
2016-01-05 11:48:51 [scrapy] INFO: Enabled downloader middlewares: HttpAuthMiddleware, DownloadTimeoutMiddleware, UserAgentMiddlewar
e, RetryMiddleware, DefaultHeadersMiddleware, MetaRefreshMiddleware, HttpCompressionMiddleware, RedirectMiddleware, CookiesMiddlewar
e, ChunkedTransferMiddleware, DownloaderStats
2016-01-05 11:48:51 [scrapy] INFO: Enabled spider middlewares: HttpErrorMiddleware, OffsiteMiddleware, RefererMiddleware, UrlLengthM
iddleware, DepthMiddleware
2016-01-05 11:48:51 [scrapy] INFO: Enabled item pipelines:
2016-01-05 11:48:51 [scrapy] INFO: Spider opened
2016-01-05 11:48:51 [scrapy] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2016-01-05 11:48:51 [scrapy] DEBUG: Telnet console listening on 127.0.0.1:6023
2016-01-05 11:48:52 [scrapy] DEBUG: Crawled (200) &lt;GET http://www.tm-alumni.eu/&gt; (referer: None)
2016-01-05 11:48:53 [scrapy] DEBUG: Crawled (200) &lt;POST http://www.tm-alumni.eu/authentication/index/login&gt; (referer: http://www.tm-
alumni.eu/)
2016-01-05 11:48:53 [temaSpider] DEBUG: =========Successfully logged in.=========
2016-01-05 11:48:55 [scrapy] DEBUG: Crawled (200) &lt;GET http://www.tm-alumni.eu/#/annuaire/diplomes?user_type=1&amp;filterGeo=1&amp;activatio
n_status=-1&amp;view=tromb&amp;page=1&gt; (referer: http://www.tm-alumni.eu/authentication/index/login)
2016-01-05 11:48:55 [temaSpider] DEBUG: =========Data is flowing.=========
2016-01-05 11:48:55 [temaSpider] DEBUG: http://www.tm-alumni.eu/
</code></pre>

<p>Thank you in advance for your help!</p>