Easiest way to get a http.response object in Scrapy
<p>I'm new to Scrapy and I am trying to "get the content of a web page" into a response object (if I correctly understood).</p>

<p>I am following <a href="http://doc.scrapy.org/en/latest/topics/selectors.html" rel="nofollow">http://doc.scrapy.org/en/latest/topics/selectors.html</a>,
which however works in scrapy shell. I would like to make it work in python code directly.</p>

<p>I wrote the code to scrap <a href="http://doc.scrapy.org/en/latest/_static/selectors-sample1.html" rel="nofollow">http://doc.scrapy.org/en/latest/_static/selectors-sample1.html</a></p>

<pre><code>import scrapy
from scrapy.http import HtmlResponse
URL = 'http://doc.scrapy.org/en/latest/_static/selectors-sample1.html'
response = HtmlResponse(url=URL)    
print response.selector.xpath('//title/text()')
</code></pre>

<p>and the output is</p>

<pre><code>&gt;&gt; []
</code></pre>

<p>Why Can't I get the proper value for title? It seems that HtmlResponse() is not downloading data from the web... why? How can I fix!</p>

<p>Tnx u very much!</p>

<p>Cap</p>