<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 Preprocessing
... but you can't access it! So MDR has done, it, below...
Download and unzip the data - MDR
(Don't re-run the below unless needed - it's >800Mb, and takes about 3-4 min to download)
Now the unzipping (15s, no output).
Step1:  Process the data
Step2:  Takes just under 2 min, no output.
Step3:  Looking at the vectors
Then the following function will return the word vectors as a matrix, the word list, and the mapping from word to index.
Step4:  Here's the first 25 "words" in glove.
Step5:  This is how you can look up a word vector.
Step6:  Just for fun, let's take a look at a 2d projection of the first 350 words, using T-SNE.
<ASSISTANT_TASK:>
Python Code:
import zipfile
with zipfile.ZipFile(path + "glove.6B.zip","r") as zip_ref:
    zip_ref.extractall(path)
%ls $path
Explanation: Preprocessing
... but you can't access it! So MDR has done, it, below...
Download and unzip the data - MDR
(Don't re-run the below unless needed - it's >800Mb, and takes about 3-4 min to download)
Now the unzipping (15s, no output).
End of explanation
import pickle
def get_glove(name):
    with open(path+ 'glove.' + name + '.txt', 'r') as f: lines = [line.split() for line in f]
    words = [d[0] for d in lines]
    vecs = np.stack(np.array(d[1:], dtype=np.float32) for d in lines)
    wordidx = {o:i for i,o in enumerate(words)}
    save_array(res_path+name+'.dat', vecs)
    pickle.dump(words, open(res_path+name+'_words.pkl','wb'))
    pickle.dump(wordidx, open(res_path+name+'_idx.pkl','wb'))
Explanation: Process the data
End of explanation
get_glove('6B.50d')
get_glove('6B.100d')
get_glove('6B.200d')
get_glove('6B.300d')
Explanation: Takes just under 2 min, no output.
End of explanation
def load_glove(loc):
    return (load_array(loc+'.dat'),
        pickle.load(open(loc+'_words.pkl','rb')),
        pickle.load(open(loc+'_idx.pkl','rb')))
vecs, words, wordidx = load_glove(res_path+'6B.50d')
vecs.shape
Explanation: Looking at the vectors
Then the following function will return the word vectors as a matrix, the word list, and the mapping from word to index.
End of explanation
' '.join(words[:25])
Explanation: Here's the first 25 "words" in glove.
End of explanation
def w2v(w): return vecs[wordidx[w]]
w2v('of')
Explanation: This is how you can look up a word vector.
End of explanation
## MDR: none of this seems to be needed?!
#reload(sys)
#sys.setdefaultencoding('utf8')   
tsne = TSNE(n_components=2, random_state=0)
Y = tsne.fit_transform(vecs[:500])
start=0; end=400
dat = Y[start:end]
plt.figure(figsize=(15,15))
plt.scatter(dat[:, 0], dat[:, 1])
for label, x, y in zip(words[start:end], dat[:, 0], dat[:, 1]):
    plt.text(x,y,label, color=np.random.rand(3)*0.7,
                 fontsize=10)
plt.show()
Explanation: Just for fun, let's take a look at a 2d projection of the first 350 words, using T-SNE.
End of explanation
<END_TASK>