How to use Scrapy FormRequest.from_response() method to submit a form?
<p>I am trying to parse some info from <a href="http://www.zooroyal.de/muehldorfer-pferdeleckerli-bio-bronchial?c=6259" rel="nofollow">this</a> page and then reload the page with other options from the drop-down menu called "Packungsgröße". There are 3 options, so I am trying to scrape the same page for all the options.</p>

<p>I found out that I can use the form of <code>class="upprice_config"</code> and <code>POST</code> it with the correct data. I do not know exactly which data I need to supply, so I supply all I have. Then I call the same function for parsing.</p>

<p>Any way, it does not work the way I want. It does not give me any error and seems it executes the form submission, but I do not get the required page.</p>

<p>Here is my code:</p>

<pre><code>... def parse_product(self, response):
120     sc_menu = response.xpath('//form[@class="upprice_config"]/div/select')
121         if sc_menu:
122             sel_name = sc_menu[0].xpath('@name').extract()[0]
124             for opt in sc_menu[0].xpath('option'):
126                 if opt.xpath('@selected'):
127                     selected = opt.xpath('text()').extract()[0]
129                     product['options'] = selected.strip()
130 
131                     yield product
132 
133                 # extract value
134                 value = opt.xpath('@value').extract()[0]
136                 # submit form for the new option
137                 self.submit_form(response, value, sel_name)
138         else:
139             product['options'] = ''
140 
141             yield product
142 
143 
144 # submit form for the new option
145 def submit_form(self, response, value, sel_name):
146     formdata = {'method': 'POST',
147                 'action': response.url,
148                 'value': value}
149     scrapy.http.FormRequest.from_response(response, 
150                 formdata = formdata, 
151                 clickdata = {'name': sel_name},
152                 callback = self.parse_product)
</code></pre>

<p>Can somebody point me to what I am doing wrong here. If there is a better way to do it, please share with me.</p>