Python Scrapy - Ajax Pagination Tripadvisor
<p>I'm using Python-Scrapy to scrap the reviews of tripadvisor members pages. 
Here is the url I'm using : <a href="http://www.tripadvisor.com/members/scottca075" rel="nofollow">http://www.tripadvisor.com/members/scottca075</a></p>

<p>I'm able to get the first page using scrapy. I haven't been able to get the other pages. I observed the <b><em>XHR</em></b> Request in the <b>Network</b> Tab of the browser on clicking Next button.</p>

<p>One GET and One POST request is sent:
On checking the parameters for the GET request, I see this:</p>

<pre><code>action : undefined_Other_ClickNext_REVIEWS_ALL
gaa : Other_ClickNext_REVIEWS_ALL
gal : 50
gams : 0
gapu : Vq85qQoQKjYAABktcRMAAAAh
gass : members`
</code></pre>

<p>The request url is </p>

<pre><code> `http://www.tripadvisor.com/ActionRecord?action=undefined_Other_ClickNext_REVIEWS_ALL&amp;gaa=Other_ClickNext_REVIEWS_ALL&amp;gal=0&amp;gass=members&amp;gapu=Vq8xPAoQLnMAAUutB9gAAAAJ&amp;gams=1`
</code></pre>

<p>The parameter <code>gal</code> represents the offset. Each page has 50 reviews. On moving to the second page by clicking the next button, the parameter <code>gal</code> is set to <code>50</code>. Then, <code>100,150,200..</code>and so on. </p>

<p>The data that I want is in the POST request in json format. <a href="http://i.stack.imgur.com/8szKN.png" rel="nofollow">Image of JSON data in POST request</a>. The request url on the post request is <code>http://www.tripadvisor.com/ModuleAjax?</code></p>

<p>I'm confused as to how to make the request in scrapy to get the data.
I tried using <code>FormRequest</code> as follows:</p>

<pre><code>pagination_url = "http://www.tripadvisor.com/ActionRecord"
form_date = {'action':'undefined_Other_ClickNext_REVIEWS_ALL','gaa':'Other_ClickNext_REVIEWS_ALL', 'gal':'0','gams':'0','gapu':'Vq8EngoQL3EAAJKgcx4AAAAN','gass':'members'}
FormRequest(url=self.pagination_url, formdata=formdata, callback=self.parseItem)
</code></pre>

<p>I also tried setting headers options in the FormRequest</p>

<pre><code>headers = {'Host':'www.tripadvisor.com','Referer':'http://www.tripadvisor.com/members/prizm','X-Requested-With': 'XMLHttpRequest'}
</code></pre>

<p>If someone could explain what I'm missing and point me in the right direction that would be great. I have run out of ideas.</p>

<p>And also, I'm aware that I can use selenium. But I want to know if there is a faster way to do this.</p>