<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 Demonstrating various stages of word sense disambiguation
The example below relies on a model for the German language. Usage of the toolkit for other languages is the same.
You just need to download a model for the corresponding language.
1. Loading pre-trained sense vectors
To test with word sense embeddings you can use a pretrained model (sense vectors and sense probabilities). These sense         vectors were induced from Wikipedia using word2vec similarities between words in ego-networks. Sense probabilities are       stored in a separate file which is located next to the file with sense vectors.
Step1:  2. Getting the list of senses of a word
Probabilities of senses will be loaded automatically if placed in the same folder as sense vectors and named according to the  same scheme as our pretrained files. To examine how many senses were learned for a word call get_senses funcion
Step2:  3. Sense aware nearest neighbors
The function returns a list of sense names with probabilities for each sense. As one can see, our model has learned two senses for the word "ключ".
To understand which word sense is represented with a sense vector use most_similar function
Step3:  4. Word sense disambiguation
Step4:  Then initialise the WSD object with sense and word vectors
Step5:  The settings have the following meaning
Step6:  Putting the four steps described above together
An example of words sense induction, in this case for the English language
Step7:  SDEWaC corpus
Step8:  Word sense induction exepriment for the Russian language
<ASSISTANT_TASK:>
Python Code:
import sensegram
# see README for model download information
sense_vectors_fpath = "model/dewiki.txt.clusters.minsize5-1000-sum-score-20.sense_vectors"
sv = sensegram.SenseGram.load_word2vec_format(sense_vectors_fpath, binary=False)
Explanation: Demonstrating various stages of word sense disambiguation
The example below relies on a model for the German language. Usage of the toolkit for other languages is the same.
You just need to download a model for the corresponding language.
1. Loading pre-trained sense vectors
To test with word sense embeddings you can use a pretrained model (sense vectors and sense probabilities). These sense         vectors were induced from Wikipedia using word2vec similarities between words in ego-networks. Sense probabilities are       stored in a separate file which is located next to the file with sense vectors.
End of explanation
word = "Hund"
sv.get_senses(word)
Explanation: 2. Getting the list of senses of a word
Probabilities of senses will be loaded automatically if placed in the same folder as sense vectors and named according to the  same scheme as our pretrained files. To examine how many senses were learned for a word call get_senses funcion:
End of explanation
word = "Hund"
for sense_id, prob in sv.get_senses(word):
    print(sense_id)
    print("="*20)
    for rsense_id, sim in sv.wv.most_similar(sense_id):
        print("{} {:f}".format(rsense_id, sim))
    print("\n")
Explanation: 3. Sense aware nearest neighbors
The function returns a list of sense names with probabilities for each sense. As one can see, our model has learned two senses for the word "ключ".
To understand which word sense is represented with a sense vector use most_similar function:
End of explanation
from gensim.models import KeyedVectors
word_vectors_fpath = "model/dewiki.txt.word_vectors"
wv = KeyedVectors.load_word2vec_format(word_vectors_fpath, binary=False, unicode_errors="ignore")
Explanation: 4. Word sense disambiguation: loading word embeddings
To use our word sense disambiguation mechanism you also need word vectors or context vectors, depending on the dismabiguation  strategy. Those word are located in the model directory and has the extension .vectors.
Our WSD mechanism is based on word similarities (sim) and requires word vectors to represent context words. In following we provide a disambiguation example using similarity strategy.
First, load word vectors using gensim library:
End of explanation
from wsd import WSD
wsd_model = WSD(sv, wv, window=5, method='sim', filter_ctx=3)
Explanation: Then initialise the WSD object with sense and word vectors:
End of explanation
word = "Hund"
context = "Die beste Voraussetzung für die Hund-Katze-Freundschaft ist, dass keiner von beiden in der Vergangenheit unangenehme Erlebnisse mit der anderen Gattung hatte. Am einfachsten ist die ungleiche WG, wenn sich zwei Jungtiere ein Zuhause teilen. Bei erwachsenen Tieren ist es einfacher, wenn sich Miezi in Bellos Haushalt einnistet – nicht umgekehrt, da Hunde Rudeltiere sind. Damit ein Hund das Kätzchen aber auch als Rudelmitglied sieht und nicht als Futter sollten ein paar Regeln beachtet werden"
wsd_model.dis_text(context, word, 0, 4)
Explanation: The settings have the following meaning: it will extract at most window*2 words around the target word from the  sentence as context and it will use only three most discriminative context words for disambiguation.
Now you can disambiguate the word "table" in the sentence "They bought a table and chairs for kitchen" using       dis_text function. As input it takes a sentence with space separated tokens, a target word, and start/end indices of the target word in the given sentence.
End of explanation
import sensegram
from wsd import WSD
from gensim.models import KeyedVectors
# Input data and paths (see README for model download information)
sense_vectors_fpath = "model/wiki.txt.clusters.minsize5-1000-sum-score-20.sense_vectors"
word_vectors_fpath = "model/wiki.txt.word_vectors"
context_words_max = 3 # change this paramters to 1, 2, 5, 10, 15, 20 : it may improve the results
context_window_size = 5 # this parameters can be also changed during experiments 
word = "python"
context = "Python is an interpreted high-level programming language for general-purpose programming. Created by Guido van Rossum and first released in 1991, Python has a design philosophy that emphasizes code readability, notably using significant whitespace."
ignore_case = True
lang = "en" # to filter out stopwords
# Load models (takes long time)
sv = sensegram.SenseGram.load_word2vec_format(sense_vectors_fpath, binary=False)
wv = KeyedVectors.load_word2vec_format(word_vectors_fpath, binary=False, unicode_errors="ignore")
# Play with the model (is quick)
print("Probabilities of the senses:\n{}\n\n".format(sv.get_senses(word, ignore_case=ignore_case)))
for sense_id, prob in sv.get_senses(word, ignore_case=ignore_case):
    print(sense_id)
    print("="*20)
    for rsense_id, sim in sv.wv.most_similar(sense_id):
        print("{} {:f}".format(rsense_id, sim))
    print("\n")
# Disambiguate a word in a context
wsd_model = WSD(sv, wv, window=context_window_size, lang=lang,
                filter_ctx=context_words_max, ignore_case=ignore_case)    
print(wsd_model.disambiguate(context, word))
Explanation: Putting the four steps described above together
An example of words sense induction, in this case for the English language
End of explanation
import sensegram
from wsd import WSD
from gensim.models import KeyedVectors
# Input data and paths
sense_vectors_fpath = "model/sdewac-v3.corpus.clusters.minsize5-1000-sum-score-20.sense_vectors"
word_vectors_fpath = "model/sdewac-v3.corpus.word_vectors"
context_words_max = 3 # change this paramters to 1, 2, 5, 10, 15, 20 : it may improve the results
context_window_size = 5 # this parameters can be also changed during experiments 
word = "Maus"
context = "Die Maus ist ein Eingabegerät (Befehlsgeber) bei Computern. Der allererste Prototyp wurde 1963 nach Zeichnungen von Douglas C. Engelbart gebaut; seit Mitte der 1980er Jahre bildet die Maus für fast alle Computertätigkeiten zusammen mit dem Monitor und der Tastatur eine der wichtigsten Mensch-Maschine-Schnittstellen. Die Entwicklung grafischer Benutzeroberflächen hat die Computermaus zu einem heute praktisch an jedem Desktop-PC verfügbaren Standardeingabegerät gemacht."
ignore_case = True
# Load models (takes long time)
sv = sensegram.SenseGram.load_word2vec_format(sense_vectors_fpath, binary=False)
wv = KeyedVectors.load_word2vec_format(word_vectors_fpath, binary=False, unicode_errors="ignore")
# Play with the model (is quick)
print("Probabilities of the senses:\n{}\n\n".format(sv.get_senses(word, ignore_case=ignore_case)))
for sense_id, prob in sv.get_senses(word, ignore_case=ignore_case):
    print(sense_id)
    print("="*20)
    for rsense_id, sim in sv.wv.most_similar(sense_id):
        print("{} {:f}".format(rsense_id, sim))
    print("\n")
# Disambiguate a word in a context
wsd_model = WSD(sv, wv, window=context_window_size, lang="de",
                filter_ctx=context_words_max, ignore_case=ignore_case)    
print(wsd_model.disambiguate(context, word))
Explanation: SDEWaC corpus
End of explanation
import sensegram
from wsd import WSD
from gensim.models import KeyedVectors
# Input data and paths
sense_vectors_fpath = "model/wikipedia-ru-2018.txt.clusters.minsize5-1000-sum-score-20.sense_vectors"
word_vectors_fpath = "model/wikipedia-ru-2018.txt.word_vectors"
max_context_words = 3 # change this paramters to 1, 2, 5, 10, 15, 20 : it may improve the results
context_window_size = 20 # this parameters can be also changed during experiments 
word = "ключ"
context = "Ключ — это секретная информация, используемая криптографическим алгоритмом при зашифровании/расшифровании сообщений, постановке и проверке цифровой подписи, вычислении кодов аутентичности (MAC). При использовании одного и того же алгоритма результат шифрования зависит от ключа. Для современных алгоритмов сильной криптографии утрата ключа приводит к практической невозможности расшифровать информацию."
ignore_case = True
lang = "ru" # to filter out stopwords
# Load models (takes long time)
# wv = KeyedVectors.load_word2vec_format(word_vectors_fpath, binary=False, unicode_errors="ignore")
# sv = sensegram.SenseGram.load_word2vec_format(sense_vectors_fpath, binary=False)
# Play with the model (is quick)
print("Probabilities of the senses:\n{}\n\n".format(sv.get_senses(word, ignore_case=ignore_case)))
for sense_id, prob in sv.get_senses(word, ignore_case=ignore_case):
    print(sense_id)
    print("="*20)
    for rsense_id, sim in sv.wv.most_similar(sense_id):
        print("{} {:f}".format(rsense_id, sim))
    print("\n")
# Disambiguate a word in a context
wsd_model = WSD(sv, wv, window=context_window_size, lang=lang,
                max_context_words=max_context_words, ignore_case=ignore_case)    
print(wsd_model.disambiguate(context, word))
###########################
from pandas import read_csv 
# you can download the WSI evaluation dataset with 'git clone https://github.com/nlpub/russe-wsi-kit.git'
wikiwiki_fpath = "../russe-wsi-kit/data/main/wiki-wiki/train.csv"
activedict_fpath = "../russe-wsi-kit/data/main/active-dict/test.csv"
btsrnc_fpath = "../russe-wsi-kit/data/main/bts-rnc/test.csv"
def evaluate(dataset_fpath):
    output_fpath = dataset_fpath + ".pred.csv"
    df = read_csv(dataset_fpath, sep="\t", encoding="utf-8")
    for i, row in df.iterrows():
        sense_id, _ = wsd_model.disambiguate(row.context, row.word)
        df.loc[i, "predict_sense_id"] = sense_id
    df.to_csv(output_fpath, sep="\t", encoding="utf-8")
    print("Output:", output_fpath)
    
    return output_fpath
evaluate(wikiwiki_fpath)
evaluate(btsrnc_fpath)
evaluate(activedict_fpath)
%load_ext autoreload
%autoreload 2
Explanation: Word sense induction exepriment for the Russian language
End of explanation
<END_TASK>