<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 Table of Contents
<p><div class="lev1 toc-item"><a href="#Preparations" data-toc-modified-id="Preparations-1"><span class="toc-item-num">1&nbsp;&nbsp;</span>Preparations</a></div><div class="lev2 toc-item"><a href="#Get-fulltext-" data-toc-modified-id="Get-fulltext--11"><span class="toc-item-num">1.1&nbsp;&nbsp;</span>Get fulltext </a></div><div class="lev2 toc-item"><a href="#Segment-source-text" data-toc-modified-id="Segment-source-text-12"><span class="toc-item-num">1.2&nbsp;&nbsp;</span>Segment source text</a></div><div class="lev2 toc-item"><a href="#Read-segments-into-a-variable-" data-toc-modified-id="Read-segments-into-a-variable--13"><span class="toc-item-num">1.3&nbsp;&nbsp;</span>Read segments into a variable </a></div><div class="lev2 toc-item"><a href="#Tokenising-" data-toc-modified-id="Tokenising--14"><span class="toc-item-num">1.4&nbsp;&nbsp;</span>Tokenising </a></div><div class="lev2 toc-item"><a href="#Stemming-/-Lemmatising-" data-toc-modified-id="Stemming-/-Lemmatising--15"><span class="toc-item-num">1.5&nbsp;&nbsp;</span>Stemming / Lemmatising </a></div><div class="lev2 toc-item"><a href="#Eliminate-Stopwords-" data-toc-modified-id="Eliminate-Stopwords--16"><span class="toc-item-num">1.6&nbsp;&nbsp;</span>Eliminate Stopwords </a></div><div class="lev1 toc-item"><a href="#Characterise-passages
Step1:  Segment source text<a name="SegmentSourceText"></a>
Next, as mentioned above, we want to associate information with only passages of the text, not the text as a whole. Therefore, the text has to be segmented. The one single file is being split into meaningful smaller chunks. What exactly constitutes a meaningful chunk -- a chapter, an article, a paragraph etc. -- cannot be known independently of the text in question and of the research questions. Therefore, it is suggested that the scholar either splits the text manually or inserts some symbols that otherwise do not appear in the text. Then, processing tools can identify these and split the file accordingly. For keeping things neat and orderly, the resulting files should be saved in a directory of their own...
Here, I am splitting the file arbitrarily every 80 lines... Note though, that this leads to a rather unusual condition
Step2:  Read segments into a variable <a name="ReadSegmentsIntoVariable"></a>
From the segments, we rebuild our corpus, iterating through them and reading them into another variable (which now stores, technically speaking, a list of strings).
Step3:  Now we should have 45 strings in the variable corpus to play around with
Step4:  For a quick impression, let's see the opening 500 characters of an arbitrary one of them
Step5:  Tokenising <a name="Tokenising"></a>
"Tokenising" means splitting the long lines of the input into single words. Since we are dealing with plain latin, we can use the default split method which relies on spaces to identify word boundaries. (In languages like Japanese or scripts like Arabic, this is more difficult.) Note that we do not compensate for words that are hyphenated/split across lines here!
Step6:  For our examples, let's have a look at (the first 50 words of) an arbitrary one of those segments
Step7:  Already, we can have a first go at finding the most frequent words for a segment. (For this we use a simple library of functions that we import by the name of 'collections'.)
Step8:  Perhaps now is a good opportunity for a small excursus. What we have printed in the last code is a series of pairs
Step9:  Looks better now, doesn't it?
Stemming / Lemmatising <a name="StemmingLemmatising"></a>
Next, since we prefer to count different word forms as one and the same "lemma", we need to do a step called "lemmatisation". In languages like English, that are not strongly inflected, one can get away with "stemming", i.e. just eliminating the ending of words
Step10:  So, we can again build a dictionary of key-value pairs associating all the lemmata ("values") with their wordforms ("keys")
Step11:  Again, a quick test
Step12:  Now we can use this dictionary to build a new list of words, where only lemmatised forms occur
Step13:  As you can see, the original text is lost now from the data that we are currently working with (unless we add another dimension to our lemmatised variable which can keep the original word form). But let us see if something in the 10 most frequent words has changed
Step14:  Yes, things have changed
Step15:  Now let's try and suppress the stopwords in the segments...
Step16:  With this, we can already create a first "profile" of our first 4 segments
<ASSISTANT_TASK:>
Python Code:
bigsourcefile = 'TextProcessing_2017/W0013.orig.txt'         # This is the path to our file
input = open(bigsourcefile, encoding='utf-8').readlines()   # We use a variable 'input' for
                                                            # keeping its contents.
input[:10]                                     # Just for information,
                                               # let's see the first 10 lines of the file.
Explanation: Table of Contents
<p><div class="lev1 toc-item"><a href="#Preparations" data-toc-modified-id="Preparations-1"><span class="toc-item-num">1&nbsp;&nbsp;</span>Preparations</a></div><div class="lev2 toc-item"><a href="#Get-fulltext-" data-toc-modified-id="Get-fulltext--11"><span class="toc-item-num">1.1&nbsp;&nbsp;</span>Get fulltext </a></div><div class="lev2 toc-item"><a href="#Segment-source-text" data-toc-modified-id="Segment-source-text-12"><span class="toc-item-num">1.2&nbsp;&nbsp;</span>Segment source text</a></div><div class="lev2 toc-item"><a href="#Read-segments-into-a-variable-" data-toc-modified-id="Read-segments-into-a-variable--13"><span class="toc-item-num">1.3&nbsp;&nbsp;</span>Read segments into a variable </a></div><div class="lev2 toc-item"><a href="#Tokenising-" data-toc-modified-id="Tokenising--14"><span class="toc-item-num">1.4&nbsp;&nbsp;</span>Tokenising </a></div><div class="lev2 toc-item"><a href="#Stemming-/-Lemmatising-" data-toc-modified-id="Stemming-/-Lemmatising--15"><span class="toc-item-num">1.5&nbsp;&nbsp;</span>Stemming / Lemmatising </a></div><div class="lev2 toc-item"><a href="#Eliminate-Stopwords-" data-toc-modified-id="Eliminate-Stopwords--16"><span class="toc-item-num">1.6&nbsp;&nbsp;</span>Eliminate Stopwords </a></div><div class="lev1 toc-item"><a href="#Characterise-passages:-TF/IDF" data-toc-modified-id="Characterise-passages:-TF/IDF-2"><span class="toc-item-num">2&nbsp;&nbsp;</span>Characterise passages: TF/IDF</a></div><div class="lev1 toc-item"><a href="#Find-similar-passages:-Clustering" data-toc-modified-id="Find-similar-passages:-Clustering-3"><span class="toc-item-num">3&nbsp;&nbsp;</span>Find similar passages: Clustering</a></div><div class="lev1 toc-item"><a href="#Topic-Modelling" data-toc-modified-id="Topic-Modelling-4"><span class="toc-item-num">4&nbsp;&nbsp;</span>Topic Modelling</a></div><div class="lev1 toc-item"><a href="#Manual-Annotation" data-toc-modified-id="Manual-Annotation-5"><span class="toc-item-num">5&nbsp;&nbsp;</span>Manual Annotation</a></div><div class="lev1 toc-item"><a href="#Cope-with-different-languages" data-toc-modified-id="Cope-with-different-languages-6"><span class="toc-item-num">6&nbsp;&nbsp;</span>Cope with different languages</a></div><div class="lev1 toc-item"><a href="#Further-information" data-toc-modified-id="Further-information-7"><span class="toc-item-num">7&nbsp;&nbsp;</span>Further information</a></div>
**Text Processing**
This is an introduction to some algorithms used in text analysis. While I cannot define **what questions** a scholar can ask, I can and do describe here **the kind of information** about text that some popular methods deliver. From this, you need to draw on your own research interests and creativity...
I will describe methods of finding words that are characteristic for a certain passage ("tf/tdf"), constructing fingerprints for passages that go beyond the most significant words ("word vectors"), group passages according to their similarity ("clustering"), and forming an idea about different contexts being treated in a passage ("topic modelling"). Of course, an important resource in text analysis is the hermeneutic interpretation of the scholar herself, so I will present a method of adding manual annotations to the text, and finally I will also say something about possible approaches to working across languages.
This page will *not* cover stylistic analyses ("stylometry") and typical neighborship relations between words ("collocation", "word2vec"). Maybe these can be discussed at another occasion and on another page.
For many of the steps discussed on this page there are ready-made tools and libraries, often with easy interfaces. But first, it is important to understand what these tools are actually doing and how their results are affected by the selection of parameters (that one can or cannot modify). And second, most of these tools expect the input to be in some particular format, say, a series of plaintext files in their own directory. So, by understanding the process, you should be better prepared to provide your text to the tools in the most productive way. Finally, it is important to be aware of what information has been **lost** at which point in the process. If the research requires so, one can then either look for a different tool or approach to this step (e.g. using an additional dimension in the list of words to keep both original and regularized word forms, or to remember the position of the current token in the original text), or one can compensate for the data loss (e.g. offering a lemmatised search to find occurrences after the analysis returns only normalised word forms)...
# Preparations
As indicated above, before doing maths, language processing tools normally expect their input to be in a certain format. First of all, you have to have an input in the first place: Therefore, a scholar wishing to experiment with such methods should avail herself of the text that should be studied, as a full transcription. This can be done by transcribing it herself, using transcriptions that are available from elsewhere, or even from OCR. (Although in the latter case, the results depend of course on the quality of the OCR output.) Second, many tools get tripped up when formatting or bibliographical metainformation is included in their input. And since the approaches presented here are not concerned with a digital edition or any other form of true representation of the source, *markup* (e.g. for bold font, heading or note elements) should be *suppressed*. (Other tools accept marked up text and strip the formatting internally.)
For another detail regarding these plain text files, we have to make a short excursus, because even with plain text, there are some important aspects to consider: As you surely know, computers understand number only and as you probably also know, the first standards to encode alphanumeric characters, like ASCII, in numbers were designed for teleprinters and the reduced character set of the english language. When more extraordinary characters, like *Umlauts* or *accents* were to be encoded, one had to rely on extra rules, of which - unfortunately - there have been quite a lot. These are called "encodings" and one of the more important set of such rules are the windows encodings (e.g. CP-1252), another one is called Latin-9/ISO 8859-15 (it differs from the older Latin-1 encoding among others by including the Euro sign). Maybe you have seen web pages with garbled *Umlauts* or other special characters, then that was probably because your browser interpreted the numbers according to an encoding different from the one that the webpage author used. Anyway, the point here is that there is another standard encompassing virtually all the special signs from all languages and for a few years now, it is also supported quite well by operating systems, programming languages and linguistic tools. This standard is called "Unicode" and the encoding you want to use is called *utf-8*. So when you export or import your texts, try to make sure that this is what is used. ([Here](https://unicode-table.com/) is a webpage with the complete unicode table - it is loaded incrementally, so make sure to scroll down. But on the other hand, it is so extensive that you don't want to scroll through all the table...)
Also, you should consider whether or not you can replace *abbreviations* with their expanded versions. While at some points (e.g. when lemmatising), you can associate expansions to abbreviations, the whole processing is easier when words in the text are indeed words, and periods are rather sentence punctuation than abbreviation signs. Of course, this also depends on the effort you can spend on the text...
This section describes how the plaintext can further be prepared for analyses: E.g. if you want to process the *distribution* of words in the text, the processing method has to have some notion of different places in the text -- normally you want to manage words not according to their absolute position in the whole work (say, the 6.349th word and the 3.100th), but according to their occurrence in a particular section (say, in the third chapter, without caring too much whether it is in the 13th or in the 643th position in this chapter). So, you partition the text into meaningful segments which you can then label, compare etc.
Other preparatory work includes suppressing stopwords (like "the", "is", "of" in english) or making the tools manage different forms of the same word or different historical writings identically. Here is what falls under this category:
  1. [Get fulltext](#GetFulltext)
  2. [Segment source text](#SegmentSourceText) 
  3. [Read segments into Variable/List](#ReadSegmentsIntoVariable)
  4. [Tokenising](#Tokenising)
  5. [Stemming/Lemmatising](#StemmingLemmatising)
  6. [Eliminate stopwords](#EliminateStopwords)
## Get fulltext <a name="GetFulltext"></a>
For the examples given on this page, I have loaded a plaintext export of Francisco de Vitoria's "Relectiones" from the School of Salamanca's project, available as one single file at this URL: [http://api.salamanca.school/txt/works.W0013.orig]. I have saved this to the file **TextProcessing_2017/W0013.orig.txt**.
End of explanation
splitLen = 80                 # 80 lines per file
outputBase = 'TextProcessing_2017/segment'  # source/segment.1.txt, source/segment.2.txt, etc.
count = 0                     # initialise some variables. 
at    = 0
dest  = None                  # this later takes our destination files
for line in input:
    if count % splitLen == 0:
        if dest: dest.close()
        dest = open(outputBase + '.' + str(at) + '.txt', encoding='utf-8', mode='w')   # 'w' is for writing: here we open the file the current segment is being written to
        at += 1
    dest.write(line.strip())
    count += 1
print(str(at - 1) + ' files written.')
Explanation: Segment source text<a name="SegmentSourceText"></a>
Next, as mentioned above, we want to associate information with only passages of the text, not the text as a whole. Therefore, the text has to be segmented. The one single file is being split into meaningful smaller chunks. What exactly constitutes a meaningful chunk -- a chapter, an article, a paragraph etc. -- cannot be known independently of the text in question and of the research questions. Therefore, it is suggested that the scholar either splits the text manually or inserts some symbols that otherwise do not appear in the text. Then, processing tools can identify these and split the file accordingly. For keeping things neat and orderly, the resulting files should be saved in a directory of their own...
Here, I am splitting the file arbitrarily every 80 lines... Note though, that this leads to a rather unusual condition: all segments are of (roughly) the same length. When counting words and assessing their relative "importance", if a word occurs twice in a very short passage, this is more telling about the passage than if the passage was very, very long. Later, we will see ways to compensate for the normal variance in passage length.
End of explanation
import sys
import glob
import errno
path     = 'TextProcessing_2017'
filename = 'segment.'
suffix   = '.txt'
corpus   = []
for i in range(0, at - 1):
    try:
        with open(path + '/' + filename + str(i) + suffix, encoding='utf-8') as f:
            corpus.append(f.read())
            f.close()
    except IOError as exc:
        if exc.errno != errno.EISDIR: # Do not fail if a directory is found, just ignore it.
            raise # Propagate other kinds of IOError.
Explanation: Read segments into a variable <a name="ReadSegmentsIntoVariable"></a>
From the segments, we rebuild our corpus, iterating through them and reading them into another variable (which now stores, technically speaking, a list of strings).
End of explanation
len(corpus)
Explanation: Now we should have 45 strings in the variable corpus to play around with:
End of explanation
corpus[5][:500]
Explanation: For a quick impression, let's see the opening 500 characters of an arbitrary one of them:
End of explanation
import re
tokenised = []
for segment in corpus:
    tokenised.append(list(filter(None, (word.lower() for word in re.split('\W+', segment)))))
Explanation: Tokenising <a name="Tokenising"></a>
"Tokenising" means splitting the long lines of the input into single words. Since we are dealing with plain latin, we can use the default split method which relies on spaces to identify word boundaries. (In languages like Japanese or scripts like Arabic, this is more difficult.) Note that we do not compensate for words that are hyphenated/split across lines here!
End of explanation
print(tokenised[5][:50])
Explanation: For our examples, let's have a look at (the first 50 words of) an arbitrary one of those segments:
End of explanation
import collections
counter = collections.Counter(tokenised[5])
print(counter.most_common(10))
Explanation: Already, we can have a first go at finding the most frequent words for a segment. (For this we use a simple library of functions that we import by the name of 'collections'.):
End of explanation
import pandas as pd
df1 = pd.DataFrame.from_dict(counter, orient='index').reset_index()
df2 = df1.rename(columns={'index':'lemma',0:'count'})
df2.sort_values('count',0,False)[:10]
Explanation: Perhaps now is a good opportunity for a small excursus. What we have printed in the last code is a series of pairs: Words and their number of occurrences, sorted by the latter. Yet the display looks a bit ugly. With another library called "pandas" (for python data analysis), we can make this more intuitive. (Of course, your system must have this library installed in the first place so that we can import it in our code.):
End of explanation
wordfile_path = 'TextProcessing_2017/wordforms-lat.txt'
wordfile = open(wordfile_path, encoding='utf-8')
print (wordfile.read()[:59])
wordfile.close;                # (The semicolon suppresses the returned object in cell output)
Explanation: Looks better now, doesn't it?
Stemming / Lemmatising <a name="StemmingLemmatising"></a>
Next, since we prefer to count different word forms as one and the same "lemma", we need to do a step called "lemmatisation". In languages like English, that are not strongly inflected, one can get away with "stemming", i.e. just eliminating the ending of words: "wish", "wished", "wishing", "wishes" all can count as instances of "wish*". With Latin this is not so easy: we want to count occurrences of "legum", "leges", "lex" as one and the same word, but if we truncate after "le", we get too many hits that have nothing to do with lex at all. There are a couple of "lemmatising" tools available, we do our own with a dictionary approach...
First, we have to have a dictionary which associates all known word forms to their lemma. This also helps us with historical orthography. Suppose from some other context, we have a file "wordforms-lat.txt" at our disposal in the TextProcessing_2017 directory. Its contents looks like this:
End of explanation
lemma    = {}    # we build a so-called dictionary for the lookups
tempdict = []
wordfile = open(wordfile_path, encoding='utf-8')
for line in wordfile.readlines():
    tempdict.append(tuple(line.split('>')))
lemma = {k.strip(): v.strip() for k, v in tempdict}
wordfile.close;
print(str(len(lemma)) + ' wordforms registered.')
Explanation: So, we can again build a dictionary of key-value pairs associating all the lemmata ("values") with their wordforms ("keys"):
End of explanation
lemma['ciuicior']
Explanation: Again, a quick test: Let's see with which basic word the wordform "ciuicior" is associated, or, in other words, what value our lemma variable returns when we query for the key "ciuicior":
End of explanation
lemmatised = [[lemma[word] if word in lemma else word for word in segment] \
              for segment in tokenised]
print(lemmatised[5][:50])
Explanation: Now we can use this dictionary to build a new list of words, where only lemmatised forms occur:
End of explanation
counter2 = collections.Counter(lemmatised[5])
df1 = pd.DataFrame.from_dict(counter2, orient='index').reset_index()
df2 = df1.rename(columns={'index':'lemma',0:'count'})
df2.sort_values('count',0,False)[:10]
Explanation: As you can see, the original text is lost now from the data that we are currently working with (unless we add another dimension to our lemmatised variable which can keep the original word form). But let us see if something in the 10 most frequent words has changed:
End of explanation
stopwords_path = 'TextProcessing_2017/stopwords-lat.txt'
stopwords = open(stopwords_path, encoding='utf-8').read().splitlines()
print(str(len(stopwords)) + ' stopwords, e.g.: ' + str(stopwords[24:54]))
Explanation: Yes, things have changed: "esse/sum" has moved to the most frequent place, "non" is now counted among the "nolo" (I am not sure this makes sense, but such is the dictionary of wordforms we have used) and "potestas" has now made it from the eighth to the second place!
Eliminate Stopwords <a name="EliminateStopwords"></a>
Probably "sum/esse", "non/nolo", "in", "ad" and the like are not really very informative words. They are what one calls stopwords, and we have another list of such words that we would rather want to ignore.
End of explanation
stopped = [[item for item in lemmatised_segment if item not in stopwords] \
           for lemmatised_segment in lemmatised]
print(stopped[5][:20])
Explanation: Now let's try and suppress the stopwords in the segments...
End of explanation
counter3 = collections.Counter(stopped[0])
counter4 = collections.Counter(stopped[1])
counter5 = collections.Counter(stopped[2])
counter6 = collections.Counter(stopped[3])
df0_1 = pd.DataFrame.from_dict(counter3, orient='index').reset_index()
df0_2 = df0_1.rename(columns={'index':'lemma',0:'count'})
df1_1 = pd.DataFrame.from_dict(counter4, orient='index').reset_index()
df1_2 = df1_1.rename(columns={'index':'lemma',0:'count'})
df2_1 = pd.DataFrame.from_dict(counter5, orient='index').reset_index()
df2_2 = df2_1.rename(columns={'index':'lemma',0:'count'})
df3_1 = pd.DataFrame.from_dict(counter6, orient='index').reset_index()
df3_2 = df3_1.rename(columns={'index':'lemma',0:'count'})
print(' ')
print(' Most frequent lemmata in the first text segment')
print(df0_2.sort_values(by='count',axis=0,ascending=False)[:10])
print(' ')
print(' ')
print(' Most frequent lemmata in the second text segment')
print(df1_2.sort_values(by='count',axis=0,ascending=False)[:10])
print(' ')
print(' ')
print(' Most frequent lemmata in the third text segment')
print(df2_2.sort_values(by='count',axis=0,ascending=False)[:10])
print(' ')
print(' ')
print(' Most frequent lemmata in the fourth text segment')
print(df3_2.sort_values(by='count',axis=0,ascending=False)[:10])
Explanation: With this, we can already create a first "profile" of our first 4 segments:
End of explanation
<END_TASK>