<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 A Walk through PyStyl
Experiments in stylometry typically kick off with creating a corpus, or the collection of texts which we would like to compare. In pystyl, we use the Corpus class to represent such a text collection
Step1:  Specifying a language such as 'en' (English) is optional. Adding texts from a directory to the corpus is easy
Step2:  By default, this function assumes that all texts under this directory have been encoded in UTF-8 and that they have a .txt extension. Additionally, the syntax of the filename should be &lt;category&gt;_&lt;title&gt;.txt, where category is a label indicates e.g. a text's authorship, genre or date of composition. In our case, this directory looked like
Step3:  Our corpus currently holds these 7 texts in their raw form
Step4:  In stylometry, it typical to preprocess your corpus and remove, let's say, punctuation and lowercase texts. In pystyl, we achieve this via the preprocess() method, where the alpha_only parameter controls whether we only wish to keep alphabetic symbols
Step5:  Now, the corpus is ready to be tokenized, which is helpful if we want to start counting words
Step6:  The corpus now holds our texts in a tokenized form. Of course, the novels wildly vary in length. If we would like to split these into shorter segments of e.g. 10,000 words, we can use the segment() function.
Step7:  In stylometry, it is common to manually remove certain words, such as personal pronouns, which are more strongly tied to narrative perspective than authorial writing style. To remove these from our English texts, we can do
Step8:  As you can see, all personal pronouns have now been removed from our corpus segments. We are now ready to vectorize our corpus, which means that we will represent it as as a large two-dimensional matrix in which each row represents one of our textual segments, and each individual feature (e.g. a function word frequency) is represented in a column.
Step9:  As you can see, we have now included the 30 most common words in our corpus model (mfi stands for 'most frequent items'). These features are returned by the vectorize() method. Many other options are available; to extract the 50 most common character trigrams, for instance, you could run
Step10:  A more fundamental issue is the vectorization model we select. By default, the vectorizer will create a simple term-frequency model, which means that we will record the relative frequencies of our most frequent items in each text. In stylometry, however, there exist many more models. PyStyl also supports the tf-idf model (term frequency-inverse document frequency), which is commonly used in information retrieval to assign more weight to lower-frequency items.
Step11:  PyStyl also supports the std model which underpins Burrows's famous Delta method (and which is typically also a solid model for other applications)
Step12:  Vectorization is a foundational issue in stylometry, since it very much controls how our analyses 'see' texts. Luckily, the vectorize() method comes with many options to control this process. With the following options, we can for install control the proportion of segments to control in how many segments a feature should minimally occur (a procedure also known as 'culling')
<ASSISTANT_TASK:>
Python Code:
%load_ext autoreload 
%autoreload 1
%matplotlib inline
from pystyl.corpus import Corpus
corpus = Corpus(language='en')
Explanation: A Walk through PyStyl
Experiments in stylometry typically kick off with creating a corpus, or the collection of texts which we would like to compare. In pystyl, we use the Corpus class to represent such a text collection:
End of explanation
corpus.add_directory(directory='data/dummy')
Explanation: Specifying a language such as 'en' (English) is optional. Adding texts from a directory to the corpus is easy:
End of explanation
ls data/dummy
Explanation: By default, this function assumes that all texts under this directory have been encoded in UTF-8 and that they have a .txt extension. Additionally, the syntax of the filename should be &lt;category&gt;_&lt;title&gt;.txt, where category is a label indicates e.g. a text's authorship, genre or date of composition. In our case, this directory looked like:
End of explanation
print(corpus)
Explanation: Our corpus currently holds these 7 texts in their raw form:
End of explanation
corpus.preprocess(alpha_only=True, lowercase=True)
print(corpus)
Explanation: In stylometry, it typical to preprocess your corpus and remove, let's say, punctuation and lowercase texts. In pystyl, we achieve this via the preprocess() method, where the alpha_only parameter controls whether we only wish to keep alphabetic symbols:
End of explanation
corpus.tokenize()
print(corpus)
Explanation: Now, the corpus is ready to be tokenized, which is helpful if we want to start counting words:
End of explanation
corpus.segment(segment_size=20000)
print(corpus)
Explanation: The corpus now holds our texts in a tokenized form. Of course, the novels wildly vary in length. If we would like to split these into shorter segments of e.g. 10,000 words, we can use the segment() function.
End of explanation
corpus.remove_tokens(rm_pronouns=True)
print(corpus)
Explanation: In stylometry, it is common to manually remove certain words, such as personal pronouns, which are more strongly tied to narrative perspective than authorial writing style. To remove these from our English texts, we can do:
End of explanation
corpus.vectorize(mfi=100)
Explanation: As you can see, all personal pronouns have now been removed from our corpus segments. We are now ready to vectorize our corpus, which means that we will represent it as as a large two-dimensional matrix in which each row represents one of our textual segments, and each individual feature (e.g. a function word frequency) is represented in a column.
End of explanation
corpus.vectorize(mfi=20, ngram_type='char', ngram_size=3)
Explanation: As you can see, we have now included the 30 most common words in our corpus model (mfi stands for 'most frequent items'). These features are returned by the vectorize() method. Many other options are available; to extract the 50 most common character trigrams, for instance, you could run:
End of explanation
corpus.vectorize(mfi=30, vector_space='tf_idf')
Explanation: A more fundamental issue is the vectorization model we select. By default, the vectorizer will create a simple term-frequency model, which means that we will record the relative frequencies of our most frequent items in each text. In stylometry, however, there exist many more models. PyStyl also supports the tf-idf model (term frequency-inverse document frequency), which is commonly used in information retrieval to assign more weight to lower-frequency items.
End of explanation
corpus.vectorize(mfi=30, vector_space='tf_std')
Explanation: PyStyl also supports the std model which underpins Burrows's famous Delta method (and which is typically also a solid model for other applications):
End of explanation
corpus.vectorize(mfi=30, min_df=0.80)
from pystyl.analysis import distance_matrix, hierarchical_clustering
from pystyl.visualization import scatterplot, scatterplot_3d
from pystyl.analysis import pca
pca_coor, pca_loadings = pca(corpus, nb_dimensions=2)
scatterplot(corpus, coor=pca_coor, nb_clusters=0, loadings=pca_loadings, plot_type='static',\
           save=False, show=False, return_svg=False, outputfile="/Users/mike/Desktop/pca.pdf")
pca_coor, pca_loadings = pca(corpus, nb_dimensions=3)
scatterplot_3d(corpus, coor=pca_coor, outputfile="/Users/mike/Desktop/3d.pdf",\
           save=True, show=False, return_svg=False)
from pystyl.analysis import distance_matrix
dm = distance_matrix(corpus, 'minmax')
from pystyl.visualization import clustermap
clustermap(corpus, distance_matrix=dm, fontsize=8, color_leafs=True,\
           outputfile='/Users/mike/Desktop/cm.pdf',
          show=False, save=False, return_svg=False)
from pystyl.analysis import hierarchical_clustering
cluster_tree = hierarchical_clustering(dm, linkage='ward')
from pystyl.visualization import scipy_dendrogram, ete_dendrogram
scipy_dendrogram(corpus=corpus, tree=cluster_tree, outputfile='~/Desktop/scipy_dendrogram.pdf',\
                 fontsize=5, color_leafs=True, show=False, save=F, return_svg=False)
ete_dendrogram(corpus=corpus, tree=cluster_tree, outputfile='~/Desktop/ete_dendrogram.png',\
                 fontsize=5, color_leafs=True, show=False, save=True, return_svg=False,
              save_newick=False)
from IPython.display import Image
Image(filename='/Users/mike/Desktop/ete_dendrogram.png')
Explanation: Vectorization is a foundational issue in stylometry, since it very much controls how our analyses 'see' texts. Luckily, the vectorize() method comes with many options to control this process. With the following options, we can for install control the proportion of segments to control in how many segments a feature should minimally occur (a procedure also known as 'culling'):
End of explanation
<END_TASK>