<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 11. Semantics 1
Step1:  Import nltk and use its internal download tool to get WordNet
Step2:  Import the wordnet module
Step3:  Access synsets of a word using the synsets function
Step4:  Each synset has a definition function
Step5:  List lemmas of a synset
Step6:  List hypernyms and hyponyms of a synset
Step7:  The closure method of synsets allows us to retrieve the transitive closure of the hypernym, hyponym, etc. relations
Step8:  common_hypernyms and lowest_common_hypernyms work in relation to another synset
Step9:  To iterate through all synsets, possibly by POS-tag, use all_synsets, which returns a generator
Step10:  Excercise (optional)
Step11:  Read the embedding into a 2D numpy array. Word forms should be stored in a separate 1D array. Also create a word index, a dictionary that returns the index of each word in the embedding. Vectors should be normalized to a length of 1
Step12:  write a function that takes two words and the embedding as input and returns their cosine similarity
Step13:  Implement a function that takes a word as a parameter and returns the 5 words that are closest to it in the embedding space
Step14:  11.E3 Vector similarity in WordNet
<a id='11.E3'></a>
Use the code written in 11.E2 to analyze word groups in WordNet
Step15:  write a function that measures the similarity of two synsets based on the cosine similarity of their vectors
Step16:  Write a function that takes a synset as input and retrieves the n most similar synsets, using the above embedding
Step17:  Build the list of all words that are both in wordnet and the GloVe embedding. On a sample of 100 such words, measure Spearman correlation of synset similarity and vector similarity (use scipy.stats.spearmanr)
<ASSISTANT_TASK:>
Python Code:
!pip install nltk
Explanation: 11. Semantics 1: words - Lab excercises
11.E1 Accessing WordNet using NLTK
11.E2 Using word embeddings
11.E3 Comparing WordNet and word embeddings
11.E1 Accessing WordNet using NLTK
<a id='11.E1'></a>
NLTK (Natural Language Toolkit) is a python library for accessing many NLP tools and resources. The NLTK WordNet interface is described here: http://www.nltk.org/howto/wordnet.html
The NLTK python package can be installed using pip:
End of explanation
import nltk
nltk.download('wordnet')
Explanation: Import nltk and use its internal download tool to get WordNet:
End of explanation
from nltk.corpus import wordnet as wn
Explanation: Import the wordnet module:
End of explanation
club_synsets = wn.synsets('club')
print(club_synsets)
Explanation: Access synsets of a word using the synsets function:
End of explanation
for synset in club_synsets:
    print("{0}\t{1}".format(synset.name(), synset.definition()))
dog = wn.synsets('dog')[0]
dog.definition()
Explanation: Each synset has a definition function:
End of explanation
dog.lemmas()
Explanation: List lemmas of a synset:
End of explanation
dog.hypernyms()
dog.hyponyms()
Explanation: List hypernyms and hyponyms of a synset
End of explanation
list(dog.closure(lambda s: s.hypernyms()))
Explanation: The closure method of synsets allows us to retrieve the transitive closure of the hypernym, hyponym, etc. relations:
End of explanation
cat = wn.synsets('cat')[0]
dog.lowest_common_hypernyms(cat)
dog.common_hypernyms(cat)
dog.path_similarity(cat)
Explanation: common_hypernyms and lowest_common_hypernyms work in relation to another synset:
End of explanation
wn.all_synsets(pos='n')
for c, noun in enumerate(wn.all_synsets(pos='n')):
    if c > 5:
        break
    print(noun.name())
Explanation: To iterate through all synsets, possibly by POS-tag, use all_synsets, which returns a generator:
End of explanation
!wget http://sandbox.hlt.bme.hu/~recski/stuff/glove.6B.50d.txt.gz
!gunzip -f glove.6B.50d.txt.gz
Explanation: Excercise (optional): use WordNet to implement the "Guess the category" game: the program lists lemmas that all share a hypernym, which the user has to guess.
11.E2 Using word embeddings
<a id='11.E2'></a>
Download and extract the word embedding glove.6B, which was trained on 6 billion words of English text using the GloVe algorithm.
End of explanation
import numpy as np
# words, word_index, emb = read_embedding('glove.6B.50d.txt')
# emb = normalize_embedding(emb)
Explanation: Read the embedding into a 2D numpy array. Word forms should be stored in a separate 1D array. Also create a word index, a dictionary that returns the index of each word in the embedding. Vectors should be normalized to a length of 1
End of explanation
# vec_sim('cat', 'dog', word_index, emb)
Explanation: write a function that takes two words and the embedding as input and returns their cosine similarity
End of explanation
# print(nearest_n('dog', words, word_index, emb))
# print(nearest_n('king', words, word_index, emb))
Explanation: Implement a function that takes a word as a parameter and returns the 5 words that are closest to it in the embedding space
End of explanation
# synset_emb = embed_synsets(words, word_index, emb)
Explanation: 11.E3 Vector similarity in WordNet
<a id='11.E3'></a>
Use the code written in 11.E2 to analyze word groups in WordNet:
Create an embedding of WordNet synsets by mapping each of them to the mean of their lemmas' vectors.
End of explanation
# synset_sim(dog, cat, synset_emb)
Explanation: write a function that measures the similarity of two synsets based on the cosine similarity of their vectors
End of explanation
# nearest_n_synsets(wn.synsets('penguin')[0], synset_emb, 10)
Explanation: Write a function that takes a synset as input and retrieves the n most similar synsets, using the above embedding
End of explanation
# compare_sims(sample, synset_emb, word_index, emb)
Explanation: Build the list of all words that are both in wordnet and the GloVe embedding. On a sample of 100 such words, measure Spearman correlation of synset similarity and vector similarity (use scipy.stats.spearmanr)
End of explanation
<END_TASK>