How to crawl each and every link given on a website and collect all the text using scrapy
<p>I followed link </p>

<p><code>https://stackoverflow.com/questions/19254630/how-to-use-scrapy-to-crawl-all-items-in-a-website</code>  </p>

<p>but things does not work out for me.</p>

<p>I am trying to learn scraping data over web.I was implementing tutorial given on <a href="http://scrapy.readthedocs.io/en/latest/intro/examples.html" rel="nofollow">http://scrapy.readthedocs.io/en/latest/intro/examples.html</a></p>

<p>and able to crawl over  a given link 
here is sample code snap </p>

<pre><code>from scrapy.spiders import Spider
from scrapy.selector import Selector

from dirbot.items import Website


class DmozSpider(Spider):
    name = "dmoz"
    allowed_domains = ["dmoz.org"]
    start_urls = [
        "http://www.dmoz.org/Computers/Programming/Languages/Python/Books/",
        "http://www.dmoz.org/Computers/Programming/Languages/Python/Resources/",
    ]

    def parse(self, response):
        """
        The lines below is a spider contract. For more info see:
        http://doc.scrapy.org/en/latest/topics/contracts.html

        @url http://www.dmoz.org/Computers/Programming/Languages/Python/Resources/
        @scrapes name
        """
        sel = Selector(response)
        sites = sel.xpath('//ul[@class="directory-url"]/li')
        items = []

        for site in sites:
            item = Website()
            item['name'] = site.xpath('a/text()').extract()
            item['url'] = site.xpath('a/@href').extract()
            item['description'] = site.xpath('text()').re('-\s[^\n]*\\r')
            items.append(item)

        return items
</code></pre>

<p>and Code snap for Item is </p>

<pre><code> from scrapy.item import Item, Field


class Website(Item):

    name = Field()
    description = Field()
    url = Field() 
</code></pre>

<p>I am able to run crawler using <code>scrapy crawl dmoz</code></p>

<p>but not able to achieve following thing </p>

<pre><code>1. All given link on website
2. Didn't get all text from all possible link 
3. Want to save them to a file 
</code></pre>

<p>Can some one guide me , for what changes need to do in my code so that i can achieve my objective ? </p>