How to get the href and associated information using scrapy?
<p>I'm new to scrapy but using python for a while. I took lesson from the <code>scrapy</code> docs along with the <code>xpath</code> selectors. Now, I would like to turn the knowledge to do a small project. I'm trying to scrap the <code>job links and the associated info like job title, location, emails (if any), phone numbers (if any)</code> from the job board <code>https://www.germanystartupjobs.com/</code> using the scrapy. </p>

<p>I have this starter code, </p>

<pre><code>import scrapy   

class GermanSpider(scrapy.Spider):

    # spider name 
    name = 'germany'

    # the first page of the website 
    start_urls= ['https://www.germanystartupjobs.com/']
    print start_urls

    def parse(self, response):
        pass 

    def parse_detail(self, response):
        pass 
</code></pre>

<p>and will run the spider <code>scrapy runspider germany</code></p>

<p>Inside the <code>parse</code> function, I would like to get the <code>href</code>s and details inside the <code>parse_detail</code> function. </p>

<p>When, I opened the mentioned page with <code>chrome</code> developer tools and inspect the listed jobs,  I see that all the jobs are inside this <code>ul</code></p>

<pre><code>&lt;ul id="job-listing-view" class="job_listings job-listings-table-bordered"&gt;
</code></pre>

<p>and then, the separates jobs are listed in the many inside <code>divs</code> of </p>

<p><code>&lt;div class="job-info-row-listing-class"&gt;</code> with associate infos, say, the <code>href</code> is provided inside <code>&lt;a href="https://www.germanystartupjobs.com/job/foodpanda-berlin-germany-2-sem-manager-mf/"&gt;</code> </p>

<p>Other <code>divs</code> provides job title, company name, location etc with <code>divs</code> such as </p>

<pre><code>          &lt;div&gt;
            &lt;h4 class="job-title-class"&gt;
              SEM Manager (m/f)            &lt;/h4&gt;
          &lt;/div&gt;

          &lt;div class="job-company-name"&gt;
            &lt;normal&gt;foodpanda&lt;normal&gt;          &lt;/normal&gt;&lt;/normal&gt;&lt;/div&gt;
          &lt;/div&gt;

  &lt;div class="location"&gt;
      &lt;div class="job-location-class"&gt;&lt;i class="glyphicon glyphicon-map-marker"&gt;&lt;/i&gt;
        Berlin, Germany          &lt;/div&gt;
    &lt;/div&gt;
</code></pre>

<p>The first step will be to get the <code>href</code> using the <code>parse</code> function and then, the associated info inside the <code>parse_details</code> using the <code>response</code>. I find that the <code>email</code> and the <code>phone</code> number only provided when you will open the links from the <code>href</code>s but the title and location is provided inside the current <code>divs</code> of the same page. </p>

<p>As I mentioned, I have okay programming skill in python, but, I struggles with the using <code>xpath</code>s even after having this <a href="https://doc.scrapy.org/en/1.0/topics/selectors.html" rel="nofollow noreferrer">tutorial</a>. How do find the links and associated info ? Some sample code with little explanation will help a lot.    </p>

<p>I try using the code</p>

<pre><code>    # firstly 
    for element in response.css("job-info-row-listing-class"):

        href = element.xpath('@href').extract()[0]
        print href
        yield scrapy.Request(href, callback=self.parse_detail)

    # secondly 
    values = response.xpath('//div[@class="job-info-row-listing-class"]//a/text()').extract()

    for v in values:
        print v


    # 
    values = response.xpath('//ul[@id="job-listing-view"]//div[@class="job-info-row-listing-class"]//a/text()').extract()
</code></pre>

<p>They seems return nothing so far after runing the spider using <code>scrapy runspider germany</code></p>