<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 XPath
XPath is short for XML Path Language which is a query language for selecting nodes in an XML document. This is very useful in webscraping because all HTML documents are a form of XML documents.
Step1:  Load HTML Code
Now I'll read the code from cell number 2 and store it in html_code. Finally we will parse that into a lxml node object.
Step2:  Using xpath to find nodes in a document
There many methods for fidning a node that you are interested in from a XML or HTML document. The first way is to write the whole path separated by forward slashes /
Reading &lt;h1&gt; tag
Step3:  To read the text inside that tag you can use the text variable.
Step4:  Another way is read the text is to use the text() function in xpath.
Step5:  Working with multiple items
xpath always returns a list. If there are no matches, it will return an empty list. If there is one match it will return a list with one item.
Step6:  We can use text() function with multiple items.
Step7:  Tag selector without full path
you can select any node in your document that matches a node selector without using the full path with a double forward slash //
Step8:  Selecting one result
You can select one result from a list using [index] after your tag selector. Make sure you use it on the tag selector and not a function selector.
Notice
Step9:  Attributes selector
In this example we have two &lt;h1&gt; tags with different css classes. We can select tags based on css classes as follows
Step10:  contains() function
I want to select all items in the first list. I could use the full class for selection or I could just use one of the classed only used in the first list with the contains() function.
Step11:  Returning attributes
What if we want to read the href attribute of the &lt;a&gt; tag to get the link. This is how you do that
Step12:  Real world example
Read the list of languages with 1M+ articles on http
<ASSISTANT_TASK:>
Python Code:
import requests
from lxml import html
%%HTML
<html>
  <body>
    <h1>Favorite Python Librarires</h1>
    <ul>
      <li>Numpy</li>
      <li>Pandas</li>
      <li>requests</li>
    </ul>
  </body>
</html>
Explanation: XPath
XPath is short for XML Path Language which is a query language for selecting nodes in an XML document. This is very useful in webscraping because all HTML documents are a form of XML documents.
End of explanation
html_code = In[2]
html_code = html_code[42:-2].replace("\\n","\n")
print(html_code)
doc = html.fromstring(html_code)
Explanation: Load HTML Code
Now I'll read the code from cell number 2 and store it in html_code. Finally we will parse that into a lxml node object.
End of explanation
title = doc.xpath("/html/body/h1")[0]
title
Explanation: Using xpath to find nodes in a document
There many methods for fidning a node that you are interested in from a XML or HTML document. The first way is to write the whole path separated by forward slashes /
Reading &lt;h1&gt; tag
End of explanation
title.text
Explanation: To read the text inside that tag you can use the text variable.
End of explanation
title = doc.xpath("/html/body/h1/text()")[0]
title
Explanation: Another way is read the text is to use the text() function in xpath.
End of explanation
item_list = doc.xpath("/html/body/ul/li")
item_list
Explanation: Working with multiple items
xpath always returns a list. If there are no matches, it will return an empty list. If there is one match it will return a list with one item.
End of explanation
doc = html.fromstring(html_code)
item_list = doc.xpath("/html/body/ul/li/text()")
item_list
Explanation: We can use text() function with multiple items.
End of explanation
doc = html.fromstring(html_code)
item_list = doc.xpath("//li/text()")
item_list
Explanation: Tag selector without full path
you can select any node in your document that matches a node selector without using the full path with a double forward slash //
End of explanation
doc = html.fromstring(html_code)
item_list = doc.xpath("/html/body/ul/li[1]/text()")
item_list
%%HTML
<html>
  <body>
    <h1 class="text-muted">Favorite Python Librarires</h1>
    <ul class="nav nav-pills nav-stacked">
      <li role="presentation"><a href="http://www.numpy.org/">Numpy</a></li>
      <li role="presentation"><a href="http://pandas.pydata.org/">Pandas</a></li>
      <li role="presentation"><a href="http://python-requests.org/">requests</a></li>
    </ul>
    <h1 class="text-success">Favorite JS Librarires</h1>
    <ul class="nav nav-tabs">
      <li role="presentation"><a href="http://getbootstrap.com/">Bootstrap</a></li>
      <li role="presentation"><a href="https://jquery.com/">jQuery</a></li>
      <li role="presentation"><a href="http://d3js.org/">d3.js</a></li>
    </ul>
</html>
html_code = In[11]
html_code = html_code[42:-2].replace("\\n","\n")
print(html_code)
doc = html.fromstring(html_code)
Explanation: Selecting one result
You can select one result from a list using [index] after your tag selector. Make sure you use it on the tag selector and not a function selector.
Notice: This is index starts from 1.
End of explanation
title = doc.xpath("/html/body/h1[@class='text-muted']/text()")[0]
title
Explanation: Attributes selector
In this example we have two &lt;h1&gt; tags with different css classes. We can select tags based on css classes as follows:
End of explanation
item_list = doc.xpath("/html/body/ul[contains(@class,'nav-stacked')]/li/a/text()")
item_list
Explanation: contains() function
I want to select all items in the first list. I could use the full class for selection or I could just use one of the classed only used in the first list with the contains() function.
End of explanation
item_list = doc.xpath("/html/body/ul[contains(@class,'nav-stacked')]/li/a/@href")
item_list
Explanation: Returning attributes
What if we want to read the href attribute of the &lt;a&gt; tag to get the link. This is how you do that:
End of explanation
response = requests.get("http://www.wikipedia.org")
doc = html.fromstring(response.content, parser=html.HTMLParser(encoding="utf-8"))
lang_list = doc.xpath("//div[@class='langlist langlist-large hlist'][1]/ul/li/a/text()")
lang_list
Explanation: Real world example
Read the list of languages with 1M+ articles on http://www.wikipedia.org/
End of explanation
<END_TASK>