Scrapy Python - How to Pass URL and retrieve URL for Scraping
<p>I have very little programming experience with python more with Java.</p>
<p>I am trying to get into python and having problems with understanding a scrapy web crawler I am trying to setup.</p>
<p>The script will scrape products etc from the site and put them into a file and recursively go through all landing domains within the site but stop at a specified depth.</p>
<p>I'm having trouble understanding how I can pass a URL exectued within a script to an example of scrapy I found.</p>
<p>Code that executes my spider:</p>
<h1>Scrappy Code is here --------------------------------</h1>
<pre><code>process = CrawlerProcess({
    'USER_AGENT': 'Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1)'
})

process.crawl(UrlScrappyRunner, domain=&quot;www.google.com&quot;)
process.start()
</code></pre>
<p>My Spider:</p>
<pre><code>class UrlScrappyRunner(scrapy.Spider):

        name = &quot;quotes&quot;

        def start_requests(self):
            urls = [
                'http://quotes.toscrape.com/page/1/',
                'http://quotes.toscrape.com/page/2/',
            ]
            for url in urls:
                yield scrapy.Request(url=url, callback=self.parse)

        def parse(self, response):
            page = response.url.split(&quot;/&quot;)[-2]
            filename = 'quotes-%s.html' % page
            with open(filename, 'wb') as f:
                f.write(response.body)
            self.log('Saved file %s' % filename)
</code></pre>
<p>Please can you let me know how to pass the domain=www.google.com to my spider so it crawls google rather than quotes.toscrape.com?</p>