<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 Introduction to Tethne
Step1:  Once again, read() accepts a string containing a path to either a single DfR dataset, or a directory containing several. Here, "DfR dataset" refers to the folder containing the file "citations.xml", and the contents of that folder.
This will take considerably more time than loading a WoS dataset. The reason is that Tethne automatically detects and parses all of the wordcount data.
Step2:  Combining DfR and WoS data
We can combine our datasets using the merge() function. First, we load our WoS data in a separate Corpus
Step3:  Both of these datasets are for the Journal of the History of Biology. But note that the WoS and DfR corpora have different numbers of Papers
Step4:  Then import merge() from tethne.readers
Step5:  We then create a new Corpus by passing both Corpus objects to merge(). If there is conflicting information in the two corpora, the first Corpus gets priority.
Step6:  merge() has combined data where possible, and discarded any duplicates in the original datasets.
Step7:  FeatureSets
Our wordcount data are represented by a FeatureSet. A FeatureSet is a description of how certain sets of elements are distributed across a Corpus. This is kind of like an inversion of an index. For example, we might be interested in which words (elements) are found in which Papers. We can think of authors as a FeatureSet, too.
All of the available FeatureSets are available in the features attribute (a dictionary) of our Corpus. We can see the available FeatureSets by inspecting its
Step8:  Note that citations and authors are also FeatureSets. In fact, the majority of network-building functions in Tethne operate on FeatureSets -- including the coauthors() and bibliographic_coupling() functions that we used in the WoS notebook.
Each FeatureSet has several attributes. The features attribute contains the distribution data itself. These data themselves are (element, value) tuples. In this case, the elements are words, and the values are wordcounts.
Step9:  The index contains our "vocabulary"
Step10:  We can use the feature_distribution() method of our Corpus to look at the distribution of words over time. In the example below I used MatPlotLib to visualize the distribution.
Step11:  If we add the argument mode='documentCounts', we get the number of documents in which 'evolutionary' occurs.
Step12:  Note that we can look how documents themselves are distributed using the distribution() method.
Step13:  So, putting these together, we can normalize our feature_distribution() data to get a sense of the relative use of the word 'evolution'.
Step14:  Topic Modeling with DfR wordcounts
Latent Dirichlet Allocation is a popular approach to discovering latent "topics" in large corpora. Many digital humanists use a software package called MALLET to fit LDA to text data. Tethne uses MALLET to fit LDA topic models.
Before we use LDA, however, we need to do some preprocessing. "Preprocessing" refers to anything that we do to filter or transform our FeatureSet prior to analysis. 
Pre-processing
Two important preprocessing steps are
Step15:  We then need to define what elements to keep, and what elements to discard. We will use a function that will evaluate whether or not a word is in our stoplist. The function should take three arguments
Step16:  We apply the stoplist using the transform() method. FeatureSets are not modified in place; instead, a new FeatureSet is generated that reflects the specified changes. We'll call the new FeatureSet 'wordcounts_filtered'.
Step17:  There should be significantly fewer words in our new "wordcounts_filtered" FeatureSet.
Step18:  The LDA topic model
Tethne provides a class called LDAModel. You should be able to import it directly from the tethne package
Step19:  Now we'll create a new LDAModel for our Corpus. The featureset_name parameter tells the LDAModel which FeatureSet we want to use. We'll use our filtered wordcounts.
Step20:  Next we'll fit the model. We need to tell MALLET how many topics to fit (the hyperparameter Z), and how many iterations (max_iter) to perform. This step may take a little while, depending on the size of your corpus.
Step21:  You can inspect the inferred topics using the model's print_topics() method. By default, this will print the top ten words for each topic.
Step22:  We can also look at the representation of a topic over time using the topic_over_time() method. In the example below we'll print the first five of the topics on the same plot.
Step23:  Generating networks from topic models
The features module in the tethne.networks subpackage contains some useful methods for visualizing topic models as networks. You can import it just like the authors or papers modules.
Step24:  The terms function generates a network of words connected on the basis of shared affinity with a topic. If two words i and j are both associated with a topic z with $\Phi(i|z) >= 0.01$ and $\Phi(j|z) >= 0.01$, then an edge is drawn between them.
Step25:  
<ASSISTANT_TASK:>
Python Code:
from tethne.readers import dfr
Explanation: Introduction to Tethne: Working with data from the Web of Science
Now that we have the basics down, in this notebook we'll begin working with data from the JSTOR Data-for-Research (DfR) portal.
The JSTOR DfR portal gives researchers access to
bibliographic data and N-grams for the entire JSTOR database.
Tethne can use DfR data to generate coauthorship networks, and to improve
metadata for Web of Science records. Tethne is also able to use
N-gram counts to add information to networks, and can interface with MALLET to perform LDA topic modeling.
Methods in Digital & Computational Humanities
This notebook is part of a cluster of learning resources developed by the Laubichler Lab and the Digital Innovation Group at Arizona State University as part of an initiative for digital and computational humanities (d+cH). For more information, see our evolving online methods course at https://diging.atlassian.net/wiki/display/DCH.
Getting Help
Development of the Tethne project is led by Erick Peirson. To get help, first check our issue tracking system on GitHub. There, you can search for questions and problems reported by other users, or ask a question of your own. You can also reach Erick via e-mail at erick.peirson@asu.edu.
Getting bibliographic data from JSTOR Data-for-Research
For the purpose of this tutorial, you can use the sample dataset from https://www.dropbox.com/s/q2jy87pmy9r6bsa/tethne_workshop_data.zip?dl=0.
Access the DfR portal at http://dfr.jstor.org/ If you don't already have an
account, you will need to create a new account.
After you've logged in, perform a search using whatever criteria you please.
When you have achieved the result that you desire, create a new dataset request.
Under the "Dataset Request" menu in the upper-right corner of the page, click
"Submit new request".
On the Download Options page, select your desired Data Type. If you do
not intend to make use of the contents of the papers themselves, then "Citations
Only" is sufficient. Otherwise, choose word counts, bigrams, etc.
Output Format should be set to XML.
Give your request a title, and set the maximum number of articles. Note that
the maximum documents allowed per request is 1,000. Setting Maximum Articles
to a value less than the number of search results will yield a random sample of
your results.
Your request should now appear in your list of Data Requests. When your
request is ready (hours to days later), you will receive an e-mail with a
download link. When downloading from the Data Requests list, be sure to use
the link in the full dataset column.
When your dataset download is complete, unzip it. The contents should look
something like those shown below.
citations.XML contains bibliographic data in XML format. The bigrams,
trigrams, wordcounts folders contain N-gram counts for each document.
If you were to open one of the XML files in the wordcounts folder, say, you would see some XML that looks like this:
```
<?xml version="1.0" encoding="UTF-8"?>
<article id="10.2307/4330482" >
    <wordcount weight="21" > of </wordcount>
    <wordcount weight="16" > the </wordcount>
    <wordcount weight="10" > university </wordcount>
    <wordcount weight="10" > a </wordcount>
    <wordcount weight="9" > s </wordcount>
    <wordcount weight="9" > d </wordcount>
    <wordcount weight="9" > harvard </wordcount>
    <wordcount weight="8" > m </wordcount>
    <wordcount weight="7" > and </wordcount>
    <wordcount weight="6" > u </wordcount>
    <wordcount weight="6" > press </wordcount>
    <wordcount weight="5" > cambridge </wordcount>
    <wordcount weight="5" > massachusetts </wordcount>
    <wordcount weight="5" > journal </wordcount>
    <wordcount weight="4" > by </wordcount>
    ...
    <wordcount weight="1" > stephen </wordcount>
    <wordcount weight="1" > liver </wordcount>
    <wordcount weight="1" > committee </wordcount>
    <wordcount weight="1" > school </wordcount>
    <wordcount weight="1" > lewontin </wordcount>
    <wordcount weight="1" > canguilhem </wordcount>
    <wordcount weight="1" > assistant </wordcount>
    <wordcount weight="1" > jay </wordcount>
    <wordcount weight="1" > state </wordcount>
    <wordcount weight="1" > morgan </wordcount>
    <wordcount weight="1" > advertising </wordcount>
    <wordcount weight="1" > animal </wordcount>
    <wordcount weight="1" > is </wordcount>
    <wordcount weight="1" > species </wordcount>
    <wordcount weight="1" > claude </wordcount>
    <wordcount weight="1" > review </wordcount>
    <wordcount weight="1" > hunt </wordcount>
    <wordcount weight="1" > founder </wordcount>
</article>
```
Each word is represented by a &lt;wordcount&gt;&lt;/wordcount&gt; tag. The "weight" attribute gives the number of times that the word occurs in the document, and the word itself is between the tags. We'll come back to this in just a moment.
Parsing DfR datasets
Just as for WoS data, there is a module in tethne.readers for working with DfR data. We can import it with:
End of explanation
dfr_corpus = dfr.read('/Users/erickpeirson/Dropbox/HSS ThatCamp Workshop/sample_data/DfR')
Explanation: Once again, read() accepts a string containing a path to either a single DfR dataset, or a directory containing several. Here, "DfR dataset" refers to the folder containing the file "citations.xml", and the contents of that folder.
This will take considerably more time than loading a WoS dataset. The reason is that Tethne automatically detects and parses all of the wordcount data.
End of explanation
from tethne.readers import wos
wos_corpus = wos.read('/Users/erickpeirson/Dropbox/HSS ThatCamp Workshop/sample_data/wos')
Explanation: Combining DfR and WoS data
We can combine our datasets using the merge() function. First, we load our WoS data in a separate Corpus:
End of explanation
len(dfr_corpus), len(wos_corpus)
Explanation: Both of these datasets are for the Journal of the History of Biology. But note that the WoS and DfR corpora have different numbers of Papers:
End of explanation
from tethne.readers import merge
Explanation: Then import merge() from tethne.readers:
End of explanation
corpus = merge(dfr_corpus, wos_corpus)
Explanation: We then create a new Corpus by passing both Corpus objects to merge(). If there is conflicting information in the two corpora, the first Corpus gets priority.
End of explanation
len(corpus)
Explanation: merge() has combined data where possible, and discarded any duplicates in the original datasets.
End of explanation
corpus.features
Explanation: FeatureSets
Our wordcount data are represented by a FeatureSet. A FeatureSet is a description of how certain sets of elements are distributed across a Corpus. This is kind of like an inversion of an index. For example, we might be interested in which words (elements) are found in which Papers. We can think of authors as a FeatureSet, too.
All of the available FeatureSets are available in the features attribute (a dictionary) of our Corpus. We can see the available FeatureSets by inspecting its:
End of explanation
corpus.features['wordcounts'].features.items()[0]    # Just show data for the first Paper.
Explanation: Note that citations and authors are also FeatureSets. In fact, the majority of network-building functions in Tethne operate on FeatureSets -- including the coauthors() and bibliographic_coupling() functions that we used in the WoS notebook.
Each FeatureSet has several attributes. The features attribute contains the distribution data itself. These data themselves are (element, value) tuples. In this case, the elements are words, and the values are wordcounts.
End of explanation
print 'There are %i words in the wordcounts featureset' % len(corpus.features['wordcounts'].index)
Explanation: The index contains our "vocabulary":
End of explanation
plt.figure(figsize=(10, 5))
plt.bar(*corpus.feature_distribution('wordcounts', 'evolutionary'))    # <-- The action.
plt.ylabel('Frequency of the word ``evolutionary`` in this Corpus')
plt.xlabel('Publication Date')
plt.show()
Explanation: We can use the feature_distribution() method of our Corpus to look at the distribution of words over time. In the example below I used MatPlotLib to visualize the distribution.
End of explanation
plt.figure(figsize=(10, 5))
plt.bar(*corpus.feature_distribution('wordcounts', 'evolutionary', mode='documentCounts'))    # <-- The action.
plt.ylabel('Documents containing ``evolutionary``')
plt.xlabel('Publication Date')
plt.show()
Explanation: If we add the argument mode='documentCounts', we get the number of documents in which 'evolutionary' occurs.
End of explanation
plt.figure(figsize=(10, 5))
plt.bar(*corpus.distribution())    # <-- The action.
plt.ylabel('Number of Documents')
plt.xlabel('Publication Date')
plt.show()
Explanation: Note that we can look how documents themselves are distributed using the distribution() method.
End of explanation
dates, N_evolution = corpus.feature_distribution('wordcounts', 'evolutionary', mode='documentCounts')
dates, N = corpus.distribution()
normalized_frequency = [f/N[i] for i, f in enumerate(N_evolution)]
plt.figure(figsize=(10, 5))
plt.bar(dates, normalized_frequency)    # <-- The action.
plt.ylabel('Proportion of documents containing ``evolutionary``')
plt.xlabel('Publication Date')
plt.show()
Explanation: So, putting these together, we can normalize our feature_distribution() data to get a sense of the relative use of the word 'evolution'.
End of explanation
from nltk.corpus import stopwords
stoplist = stopwords.words()
Explanation: Topic Modeling with DfR wordcounts
Latent Dirichlet Allocation is a popular approach to discovering latent "topics" in large corpora. Many digital humanists use a software package called MALLET to fit LDA to text data. Tethne uses MALLET to fit LDA topic models.
Before we use LDA, however, we need to do some preprocessing. "Preprocessing" refers to anything that we do to filter or transform our FeatureSet prior to analysis. 
Pre-processing
Two important preprocessing steps are:
1. Removing "stopwords" -- common words like "the", "and", "but", "for", that don't yield much insight into the contents of documents.
2. Removing words that are too common or too rare. These include typos or OCR artifacts.
We can do both of these by using the transform() method on our FeatureSet.
First, we need a stoplist. NLTK provides a great stoplist.
End of explanation
def apply_stoplist(f, v, c, dc):
    if f in stoplist or dc > 500 or dc < 3 or len(f) < 4:
        return None    # Discard the element.
    return v
Explanation: We then need to define what elements to keep, and what elements to discard. We will use a function that will evaluate whether or not a word is in our stoplist. The function should take three arguments:
f -- the feature itself (the word)
v -- the number of instances of that feature in a specific document
c -- the number of instances of that feature in the whole FeatureSet
dc -- the number of documents that contain that feature
This function will be applied to each word in each document. If it returns 0 or None, the word will be excluded. Otherwise, it should return a numeric value (in this case, the count for that document).
In addition to applying the stoplist, we'll also exclude any word that occurs in more than 500 of the documents and less than 3 documents, and is less than 4 characters in length.
End of explanation
corpus.features['wordcounts_filtered'] = corpus.features['wordcounts'].transform(apply_stoplist)
Explanation: We apply the stoplist using the transform() method. FeatureSets are not modified in place; instead, a new FeatureSet is generated that reflects the specified changes. We'll call the new FeatureSet 'wordcounts_filtered'.
End of explanation
print 'There are %i words in the wordcounts featureset' % len(corpus.features['wordcounts'].index)
print 'There are %i words in the wordcounts_filtered featureset' % len(corpus.features['wordcounts_filtered'].index)
Explanation: There should be significantly fewer words in our new "wordcounts_filtered" FeatureSet.
End of explanation
from tethne import LDAModel
Explanation: The LDA topic model
Tethne provides a class called LDAModel. You should be able to import it directly from the tethne package:
End of explanation
model = LDAModel(corpus, featureset_name='wordcounts_filtered')
Explanation: Now we'll create a new LDAModel for our Corpus. The featureset_name parameter tells the LDAModel which FeatureSet we want to use. We'll use our filtered wordcounts.
End of explanation
model.fit(Z=50, max_iter=500)
Explanation: Next we'll fit the model. We need to tell MALLET how many topics to fit (the hyperparameter Z), and how many iterations (max_iter) to perform. This step may take a little while, depending on the size of your corpus.
End of explanation
model.print_topics()
Explanation: You can inspect the inferred topics using the model's print_topics() method. By default, this will print the top ten words for each topic.
End of explanation
plt.figure(figsize=(15, 5))
for k in xrange(5):    # Generates numbers k in [0, 4].
    x, y = model.topic_over_time(k)    # Gets topic number k.
    plt.plot(x, y, label='topic {0}'.format(k), lw=2, alpha=0.7)
plt.legend(loc='best')
plt.show()    
Explanation: We can also look at the representation of a topic over time using the topic_over_time() method. In the example below we'll print the first five of the topics on the same plot.
End of explanation
from tethne.networks import topics
Explanation: Generating networks from topic models
The features module in the tethne.networks subpackage contains some useful methods for visualizing topic models as networks. You can import it just like the authors or papers modules.
End of explanation
termGraph = topics.terms(model, threshold=0.01)
termGraph.order(), termGraph.size()
termGraph.name = ''
from tethne.writers.graph import to_graphml
to_graphml(termGraph, '/Users/erickpeirson/Desktop/topic_terms.graphml')
Explanation: The terms function generates a network of words connected on the basis of shared affinity with a topic. If two words i and j are both associated with a topic z with $\Phi(i|z) >= 0.01$ and $\Phi(j|z) >= 0.01$, then an edge is drawn between them.
End of explanation
topicCoupling = topics.topic_coupling(model, threshold=0.2)
print '%i nodes and %i edges' % (topicCoupling.order(), topicCoupling.size())
to_graphml(topicCoupling, '/Users/erickpeirson/Desktop/lda_topicCoupling.graphml')
Explanation: 
End of explanation
<END_TASK>