Pass values into scrapy callback
<p>I'm trying to get started crawling and scraping a website to disk but having trouble getting the callback function working as I would like. </p>

<p>The code below will visit the start_url and find all the "a" tags on the site. For each 1 of them it will make a callback which is to save the text response to disk and use the crawerItem to store some metadata about the page.</p>

<p>I was hoping someone could help me figure out how to pass </p>

<ol>
<li>a unique id to each callback so it can be used as the filename when saving the file</li>
<li>Pass the url of the originating page so it can be added to the metadata via the Items </li>
<li>Follow the links on the child pages to go another level deeper into the site </li>
</ol>

<p>Below is my code thus far </p>

<pre><code>import scrapy
from scrapy.spiders import Rule
from scrapy.linkextractors import LinkExtractor
from mycrawler.items import crawlerItem

class CrawlSpider(scrapy.Spider):
    name = "librarycrawler"
    allowed_domains = ["example.com"]
    start_urls = [
        "http://www.example.com"
    ]

    rules = (
    Rule(LinkExtractor(),callback='scrape_page', follow=True)
)

def scrape_page(self,response):
    page_soup = BeautifulSoup(response.body,"html.parser")
    ScrapedPageTitle = page_soup.title.get_text()
    item = LibrarycrawlerItem()
    item['title'] =ScrapedPageTitle
    item['file_urls'] = response.url

    yield item
</code></pre>

<p><strong>In Settings.py</strong></p>

<pre><code>ITEM_PIPELINES = [
    'librarycrawler.files.FilesPipeline',
]
FILES_STORE = 'C:\Documents\Spider\crawler\ExtractedText'   
</code></pre>

<p><strong>In items.py</strong></p>

<pre><code>import scrapy


class LibrarycrawlerItem(scrapy.Item):
    # define the fields for your item here like:
    # name = scrapy.Field()
    title = scrapy.Field()
    Files = scrapy.Field()
</code></pre>