scrapy xpath ratemyprofessor
<p>I am new to the scrapy and already spent so much times on this easy program but I cannot figure out.  I used the chrome to inspect the x path for links of all professor in this page and used the console to test the xpath. when I put the "correct xpath" inside the spider, it won't work anymore and the size of links return 0" I am so confused now.Could anyone give me help?</p>

<pre><code>import scrapy

from scraper.items import ScraperItem
from scrapy.contrib.spiders import Rule
from scrapy.contrib.linkextractors import LinkExtractor


class scraperSpider(scrapy.Spider):
    name = "scraper"
    allowed_domains = ["www.ratemyprofessors.com"]
    start_urls = [
    "http://www.ratemyprofessors.com/search.jsp?queryoption=TEACHER&amp;queryBy=schoolDetails&amp;schoolID=1584&amp;schoolName=Utah+Valley+University&amp;dept=Accounting"
    ]



    def parse(self, response):
        # professors = []

        # create array of profile links
        profiles = response.xpath('//*[@id="mainContent"]/div[1]/div/div[5]/ul/li/a/@href').extract()
        print " size of profiles a " , len(profiles)
        for link in profiles:
            print link
</code></pre>