XPath returns empty list. Why is it ignoring targeted div element?
<p>I'm a newbee to XPath and Scrapy. I'm trying to target a node which does not have a unique class (i.e. <code>class="pubBody"</code>). </p>

<p>Already tried: 
<a href="https://stackoverflow.com/questions/28163626/xpath-not-contains-a-and-b">xpath not contains A and B</a></p>

<p>This should be a simple task but XPath just misses the second item. I am doing this from the scrapy shell. On the command prompt:</p>

<p>scrapy shell "<a href="http://www.sciencedirect.com/science/journal/00221694/" rel="nofollow noreferrer">http://www.sciencedirect.com/science/journal/00221694/</a>"</p>

<p>I am looking for the second div:</p>

<pre><code>&lt;div id="issueListHeader" class="pubBody"&gt;...&lt; /div&gt;

&lt;div class="pubBody"&gt;... &lt; /div&gt; 
</code></pre>

<p>I can only get the first but not the second. The best answers to similar questions suggested trying something like:</p>

<pre><code>hxs.xpath('//div[contains(@class,"pubBody") and not(contains(@id,"issueListHeader"))]') 
</code></pre>

<p>but this returns an empty list for some reason. Any help please? Must be missing something silly, I've tried this for days!</p>

<p>Other details:</p>

<p>Once in the scrapy shell:</p>

<pre><code>import scrapy

xs = scrapy.Selector(response)

hxs.xpath('//div[@class="pubBody"]')
</code></pre>

<p>Which works only for the first div element:</p>

<pre><code>[&lt;Selector xpath='//div[@class="pubBody"]' data='&lt;div id="issueListHeader" class="pubBody'&gt;]
</code></pre>

<p>For the failed second div element I've also tried:</p>

<pre><code>hxs.xpath('//div[@class="pubBody" and not(@id="issueListHeader")]').extract_first()

hxs.xpath('//div[starts-with(@class, "pubBody") and not(re:test(@id, "issueListHeader"))]')
</code></pre>

<p>Also directly copied the XPath from Chrome, but also returns '[]':</p>

<pre><code>hxs.xpath('//*[@id="issueList"]/div/form/div[2]')
</code></pre>