<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 Word Embeddings
Word embedding is a mapping of a word to a d-dimensional vector space.
This real valued vector representation captures semantic and syntactic features.
Polyglot offers a simple interface to load several formats of word embeddings.
Step1:  Formats
The Embedding class can read word embeddings from different sources
Step2:  Nearest Neighbors
A common way to investigate the space capture by the embeddings is to query for the nearest neightbors of any word.
Step3:  to calculate the distance between a word and the nieghbors, we can call the distances method
Step4:  The word embeddings are not unit vectors, actually the more frequent the word is the larger the norm of its own vector.
Step5:  This could be problematic for some applications and training algorithms.
We can normalize them by $L_2$ norms to get unit vectors to reduce effects of word frequency, as the following
Step6:  Vocabulary Expansion
Step7:  Not all the words are available in the dictionary defined by the word embeddings.
Sometimes it would be useful to map new words to similar ones that we have embeddings for.
Case Expansion
For example, the word GREEN is not available in the embeddings,
Step8:  we would like to return the vector that represents the word Green, to do that we apply a case expansion
Step9:  Digit Expansion
We reduce the size of the vocabulary while training the embeddings by grouping special classes of words.
Once common case of such grouping is digits.
Every digit in the training corpus get replaced by the symbol #.
For example, a number like 123.54 becomes ###.##.
Therefore, querying the embedding for a new number like 434 will result in a failure
Step10:  To fix that, we apply another type of vocabulary expansion DigitExpander.
It will map any number to a sequence of #s.
Step11:  As expected, the neighbors of the new number 434 will be other numbers
<ASSISTANT_TASK:>
Python Code:
from polyglot.mapping import Embedding
Explanation: Word Embeddings
Word embedding is a mapping of a word to a d-dimensional vector space.
This real valued vector representation captures semantic and syntactic features.
Polyglot offers a simple interface to load several formats of word embeddings.
End of explanation
embeddings = Embedding.load("/home/rmyeid/polyglot_data/embeddings2/en/embeddings_pkl.tar.bz2")
Explanation: Formats
The Embedding class can read word embeddings from different sources:
Gensim word2vec objects: (from_gensim method)
Word2vec binary/text models: (from_word2vec method)
polyglot pickle files: (load method)
End of explanation
neighbors = embeddings.nearest_neighbors("green")
neighbors
Explanation: Nearest Neighbors
A common way to investigate the space capture by the embeddings is to query for the nearest neightbors of any word.
End of explanation
embeddings.distances("green", neighbors)
Explanation: to calculate the distance between a word and the nieghbors, we can call the distances method
End of explanation
%matplotlib inline
import matplotlib.pyplot as plt
import numpy as np
norms = np.linalg.norm(embeddings.vectors, axis=1)
window = 300
smooth_line = np.convolve(norms, np.ones(window)/float(window), mode='valid')
plt.plot(smooth_line)
plt.xlabel("Word Rank"); _ = plt.ylabel("$L_2$ norm")
Explanation: The word embeddings are not unit vectors, actually the more frequent the word is the larger the norm of its own vector.
End of explanation
embeddings = embeddings.normalize_words()
neighbors = embeddings.nearest_neighbors("green")
for w,d in zip(neighbors, embeddings.distances("green", neighbors)):
  print("{:<8}{:.4f}".format(w,d))
Explanation: This could be problematic for some applications and training algorithms.
We can normalize them by $L_2$ norms to get unit vectors to reduce effects of word frequency, as the following
End of explanation
from polyglot.mapping import CaseExpander, DigitExpander
Explanation: Vocabulary Expansion
End of explanation
"GREEN" in embeddings
Explanation: Not all the words are available in the dictionary defined by the word embeddings.
Sometimes it would be useful to map new words to similar ones that we have embeddings for.
Case Expansion
For example, the word GREEN is not available in the embeddings,
End of explanation
embeddings.apply_expansion(CaseExpander)
"GREEN" in embeddings
embeddings.nearest_neighbors("GREEN")
Explanation: we would like to return the vector that represents the word Green, to do that we apply a case expansion:
End of explanation
"434" in embeddings
Explanation: Digit Expansion
We reduce the size of the vocabulary while training the embeddings by grouping special classes of words.
Once common case of such grouping is digits.
Every digit in the training corpus get replaced by the symbol #.
For example, a number like 123.54 becomes ###.##.
Therefore, querying the embedding for a new number like 434 will result in a failure
End of explanation
embeddings.apply_expansion(DigitExpander)
"434" in embeddings
Explanation: To fix that, we apply another type of vocabulary expansion DigitExpander.
It will map any number to a sequence of #s.
End of explanation
embeddings.nearest_neighbors("434")
Explanation: As expected, the neighbors of the new number 434 will be other numbers:
End of explanation
<END_TASK>