Unsupervised extraction of local and global keywords from a single text
Lida Aleksanyan Armen E. Allahverdyan
We propose an unsupervised, corpus-independent method to extract keywords
from a single text. It is based on the spatial distribution of words and the
response of this distribution to a random permutation of words. As compared to
existing methods (such as e.g. YAKE) our method has three advantages. First, it
is significantly more effective at extracting keywords from long texts. Second,
it allows inference of two types of keywords: local and global. Third, it
uncovers basic themes in texts. Additionally, our method is
language-independent and applies to short texts. The results are obtained via
human annotators with previous knowledge of texts from our database of
classical literary works (the agreement between annotators is from moderate to
substantial). Our results are supported via human-independent arguments based
on the average length of extracted content words and on the average number of
nouns in extracted words. We discuss relations of keywords with higher-order
textual features and reveal a connection between keywords and chapter
divisions.