<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 Part 1 - Crawling Wikipedia
Introduction
We're going to create a social network of characters in the Marvel Cinematic Universe. You are looking at a Jupyter notebook. Each section is a cell that can contain text or Python code. You can run a cell by selecting it, and hitting Ctrl-Enter. You will see the results of your code as it runs. Try running the cell below.
Step1:  You just ran some Python code that imports packages. Packages are pre-written Python code. The wikinetworking package contains code for crawling, text mining and graphing Wiki articles. You can access these functions in the wn object.
Getting a list of links
Our first step is getting a list of links that we want to crawl. Wikipedia has article data organized as lists of topics. (There are many that may not be listed on this page. You should search for one that works for you.) Once you find the URL of a list that contains articles you would like to crawl, paste the URL in the variable below.
Step2:  Now we can download the article and get a list of links from it.
Step3:  Many of these links may not be relevant to our topic. We can filter for links that exist inside certain types of HTML elements. You can find the type of element by inspecting a relevant link on your Wikipedia page in your browser. We can use a special type of filter called a CSS selector to get only links that are inside of specific types of elements.
Step4:  Cross referencing lists of links
Another way of filtering links is to try to cross-reference them with another list of links. First, find another URL that might shares the links you want, but excludes the links you don't want from the first list of links.
Step5:  What if you need links from a lists of lists? You can automatically crawl a list of URLs, as well. First we need to generate a list of URLs.
Step6:  And then we can crawl this list of URLs.
Step7:  Now that we have a second set of links, we can look for the intersection of the two lists. That should give us only the URLs we want.
Step8:  Let's save these links into a file so we don't have to download the data again.
Step9:  Let's also make sure we can load the data after we've saved it.
Step10:  Crawling The Links
Now that we have the relevant_links list, we just need to choose a starting article.
Step11:  We can "flatten" the data and save it for convenience.
<ASSISTANT_TASK:>
Python Code:
import wikinetworking as wn
import networkx as nx
from pyquery import PyQuery
%matplotlib inline
print "OK"
Explanation: Part 1 - Crawling Wikipedia
Introduction
We're going to create a social network of characters in the Marvel Cinematic Universe. You are looking at a Jupyter notebook. Each section is a cell that can contain text or Python code. You can run a cell by selecting it, and hitting Ctrl-Enter. You will see the results of your code as it runs. Try running the cell below.
End of explanation
url = "https://en.wikipedia.org/wiki/List_of_Marvel_Cinematic_Universe_film_actors"
print url
Explanation: You just ran some Python code that imports packages. Packages are pre-written Python code. The wikinetworking package contains code for crawling, text mining and graphing Wiki articles. You can access these functions in the wn object.
Getting a list of links
Our first step is getting a list of links that we want to crawl. Wikipedia has article data organized as lists of topics. (There are many that may not be listed on this page. You should search for one that works for you.) Once you find the URL of a list that contains articles you would like to crawl, paste the URL in the variable below.
End of explanation
links = wn.filter_links(PyQuery(url=url))
print links
Explanation: Now we can download the article and get a list of links from it.
End of explanation
selector="th"
links = wn.filter_links(PyQuery(url=url), selector=selector)
print links
Explanation: Many of these links may not be relevant to our topic. We can filter for links that exist inside certain types of HTML elements. You can find the type of element by inspecting a relevant link on your Wikipedia page in your browser. We can use a special type of filter called a CSS selector to get only links that are inside of specific types of elements.
End of explanation
another_url = ""
another_selector = ""
more_links = wn.filter_links(PyQuery(url=another_url), selector=another_selector)
print more_links
Explanation: Cross referencing lists of links
Another way of filtering links is to try to cross-reference them with another list of links. First, find another URL that might shares the links you want, but excludes the links you don't want from the first list of links.
End of explanation
url_pattern = "https://en.wikipedia.org/wiki/List_of_Marvel_Comics_characters:_"
sections = [letter for letter in 'ABCDEFGHIJKLMNOPQRSTUVWXYZ']
sections.append('0-9')
many_urls = [url_pattern + section for section in sections]
print many_urls
Explanation: What if you need links from a lists of lists? You can automatically crawl a list of URLs, as well. First we need to generate a list of URLs.
End of explanation
selector = ".hatnote"
more_links = wn.retrieve_multipage(many_urls, selector=selector, verbose=True)
Explanation: And then we can crawl this list of URLs.
End of explanation
relevant_links = wn.intersection(links, more_links)
print relevant_links
Explanation: Now that we have a second set of links, we can look for the intersection of the two lists. That should give us only the URLs we want.
End of explanation
wn.write_list(relevant_links, "relevant_links.txt")
Explanation: Let's save these links into a file so we don't have to download the data again.
End of explanation
relevant_links = wn.read_list("relevant_links.txt")
print relevant_links
Explanation: Let's also make sure we can load the data after we've saved it.
End of explanation
starting_url="/wiki/Iron_Man"
raw_crawl_data = wn.crawl(starting_url, accept=relevant_links)
import json
print json.dumps(raw_crawl_data, sort_keys=True, indent=4)
Explanation: Crawling The Links
Now that we have the relevant_links list, we just need to choose a starting article.
End of explanation
graph_data = wn.undirected_graph(raw_crawl_data)
import json
print json.dumps(graph_data, sort_keys=True, indent=4)
wn.save_dict(graph_data, "undirected_graph.json")
Explanation: We can "flatten" the data and save it for convenience.
End of explanation
<END_TASK>