<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 Building word vectors
Setup
Step1:  Data
load corpus vocab and wordidx
Step2:  load data
Step3:  Word Vectors Pre Trained
collecting biolab words
Step4:  dont need word to id dict since this is indexed with words
using biolab words for missing corpus words
Step5:  for tensorboard
Step6:  building word vectors of 200d for model
Step7:  fill in biolab vectors available
Step8:  total words not updated with training from biolab
Step9:  gcloud tensorboard serving
Step10:  for http
Step11:  write to checkpoint file
Step12:  FastText Vectors
fasttext commands used
fasttext skipgram -minCount 1 -dim 200 -epoch 10 -input corpus_text_for_fast_text.txt -output ft_wvs_200d_10e
fasttext cbow -minCount 1 -dim 200 -epoch 10 -input corpus_text_for_fast_text.txt -output ft_wvs_200d_10e
reading ft vectors
Step13:  saving all trained fast text vectors
Step14:  Viewing word vectors
<ASSISTANT_TASK:>
Python Code:
import sys
import os
import re
import collections
import itertools
import bcolz
import pickle
sys.path.append('../lib')
import gc
import random
import smart_open
import h5py
import csv
import tensorflow as tf
import gensim
import datetime as dt
from tqdm import tqdm_notebook as tqdm
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns
random_state_number = 967898
from tensorflow.python.client import device_lib
def get_available_gpus():
    local_device_protos = device_lib.list_local_devices()
    return [x.name for x in local_device_protos if x.device_type == 'GPU']
config = tf.ConfigProto()
config.gpu_options.allow_growth=True
sess = tf.Session(config=config)
get_available_gpus()
%pylab
%matplotlib inline
%load_ext autoreload
%autoreload
pd.options.mode.chained_assignment = None
pd.options.display.max_columns = 999
color = sns.color_palette()
Explanation: Building word vectors
Setup
End of explanation
corpus_vocab_list, corpus_vocab_wordidx = None, None
with open('processed/stage1/vocab_words_wordidx.pkl', 'rb') as f:
    (corpus_vocab_list, corpus_wordidx) = pickle.load(f)
print(len(corpus_vocab_list), len(corpus_wordidx))
Explanation: Data
load corpus vocab and wordidx
End of explanation
store = pd.HDFStore('processed/stage1/data_frames.h5')
train_df = store['train_df']
test_df = store['test_df']
Explanation: load data
End of explanation
from gensim.models.keyedvectors import KeyedVectors
biolab_keyed_vectors_pubmed_pmc_wiki = KeyedVectors.load_word2vec_format('external/biolab_wvs/wikipedia-pubmed-and-PMC-w2v.bin', binary=True)
biolab_words_pubmed_pmc_wiki = biolab_keyed_vectors_pubmed_pmc_wiki.vocab.keys()
biolab_words = set(biolab_words_pubmed_pmc_wiki)
len(biolab_words)
vocab_biolab = set(biolab_words) & set(vocab_words)
print (len(vocab_biolab))
vocab_biolab
vocab_not_in_biolab =set(vocab_words) - set(biolab_words)
print(len(vocab_not_in_biolab))
vocab_not_in_biolab
Explanation: Word Vectors Pre Trained
collecting biolab words
End of explanation
undesirable_ascii_characters = list(range(32))
undesirable_ascii_characters.remove(10) #keep new line since this might be used for sentence tokenizer
undesirable_charmap = dict.fromkeys(undesirable_ascii_characters)
from nltk import word_tokenize
from utils import custom_word_tokenizer, apply_custom_regx
custom_tokenized_biolab_pubmed_pmc_wiki_wv = {}
for word in vocab_biolab:
    vector = biolab_keyed_vectors_pubmed_pmc_wiki.word_vec(word)
    custom_tokenized_biolab_pubmed_pmc_wiki_wv[word.lower()] = vector
    word = word.lower().encode('ascii', 'ignore').decode('utf-8', 'ignore')
    word = str(word).translate(undesirable_charmap)
    word = apply_custom_regx(word)
    word = word.replace('\\t', '')
    for part in word_tokenize(word):
        if part in custom_tokenized_biolab_pubmed_pmc_wiki_wv:
            custom_tokenized_biolab_pubmed_pmc_wiki_wv[part] += vector
            custom_tokenized_biolab_pubmed_pmc_wiki_wv[part] /= 2
len(custom_tokenized_biolab_pubmed_pmc_wiki_wv)
Explanation: dont need word to id dict since this is indexed with words
using biolab words for missing corpus words
End of explanation
tb_vocab_size=5000
tb_vocab_biolab = list(vocab_biolab)[:tb_vocab_size]
with open("view_wvs_tb/tb_vocab.tsv", "w") as fp:
    wr = csv.writer(fp, delimiter='\n')
    wr.writerow(tb_vocab_biolab)
tb_word_vectors = np.random.randn(tb_vocab_size, 200)
for i,word in enumerate(tb_vocab_biolab):
    tb_word_vectors[i] = custom_tokenized_biolab_pubmed_pmc_wiki_wv[word]
%autoreload
from utils import visualize_embeddings_in_tensorboard
visualize_this_embedding = tb_word_vectors
print(visualize_this_embedding.shape)
metadata_path = "/home/bicepjai/Projects/dsotc/data_prep/view_wvs_tb/tb_vocab.tsv"
visualize_embeddings_in_tensorboard(visualize_this_embedding, metadata_path, "/home/bicepjai/Projects/dsotc/data_prep/view_wvs_tb")
del tb_word_vectors
Explanation: for tensorboard
End of explanation
corpus_word_vectors = np.random.randn(len(vocab_words), 200)
corpus_word_vectors.shape
Explanation: building word vectors of 200d for model
End of explanation
for word in vocab_biolab:
    dataset_corpus_word_index = vocab_wordidx[word]
    corpus_word_vectors[dataset_corpus_word_index] = custom_tokenized_biolab_pubmed_pmc_wiki_wv[word]
Explanation: fill in biolab vectors available
End of explanation
words_not_updated = set(vocab_words) - vocab_biolab
len(words_not_updated)
words_not_updated
np.save("processed/stage1/biolab_updated_wvs.npy", corpus_word_vectors)
Explanation: total words not updated with training from biolab
End of explanation
dataset_corpus_words_list = np.load("dataset_corpus_words_list.npy")
corpus_word_vectors = np.load("corpus_word_vectors.npy")
tb_vocab_size = 10000
local_tb_dir = "/home/bicepjai/Projects/ml-compete/kaggle/mskrct/data_prep_2_ft/model_wv_visualize/gcloud/"
with open(local_tb_dir+"/vocab.tsv", "wb") as fp:
    wr = csv.writer(fp, delimiter='\n')
    wr.writerow(dataset_corpus_words_list[:tb_vocab_size])
Explanation: gcloud tensorboard serving
End of explanation
# np.savetxt("model_wv_visualize/word_vectors.tsv",corpus_word_vectors[:tb_vocab_size], delimiter='\t')
Explanation: for http://projector.tensorflow.org/ vectors need to be in tsv form
End of explanation
!rm $local_tb_dir/checkpoint
!ls $local_tb_dir
from word2vec import visualize_embeddings_in_tensorboard
visualize_this_embedding = corpus_word_vectors[:tb_vocab_size]
print visualize_this_embedding.shape
# path for gcloud tensorboard
metadata_path = "/home/bicepjai/projects/tb_visual/vocab.tsv"
# metadata_path = "/home/bicepjai/Projects/ml-compete/kaggle/mskrct/data_prep_2_ft/model_wv_visualize/vocab.tsv"
visualize_embeddings_in_tensorboard(visualize_this_embedding, metadata_path, local_tb_dir)
checkpoint_txt = "model_checkpoint_path: \"/home/bicepjai/projects/tb_visual/visual_embed.ckpt-1\"\n\
all_model_checkpoint_paths: \"/home/bicepjai/projects/tb_visual/visual_embed.ckpt-1\""
with open(local_tb_dir+"/checkpoint","w") as f:
    f.seek(0)
    f.truncate()
    f.write(checkpoint_txt)
Explanation: write to checkpoint file
End of explanation
fasttext_vec_file = "processed/stage2/pretrained_word_vectors/ft_sg_200d_10e.vec"
ft_lines = None
with open(fasttext_vec_file,"r") as f:
    ft_lines = f.readlines()
print(ft_lines[0])
print(type(ft_lines), len(ft_lines))
ft_shape = tuple([int(i.strip()) for i in ft_lines[0].split()])
ft_shape
print(len(ft_lines[1].split()))
ft_lines[1]
ft_vocab_size=ft_shape[0]
ft_vocab_size
ft_word_vectors = np.random.randn(ft_vocab_size, ft_shape[1])
ft_words = []
for i, line in enumerate(ft_lines[1:]):
    str_list =line.split()
    ft_words.append(str_list[0].strip())
    vec = np.array([np.float(f) for f in str_list[1:]])
    ft_word_vectors[i] = vec
ft_word_vectors.shape
a = list(ft_words)
a.sort(key=len, reverse=True)
print(a[:10])
del a
ft_wordidx = {w:i for i,w in enumerate(ft_words)}
ft_vocab_size, len(ft_wordidx)
len(set(vocab_words) - set(ft_words))
set(vocab_words) - set(ft_words)
%autoreload
import global_utils
fasttext_vec_file="/home/bicepjai/Projects/dsotc/data_prep/processed/stage1/pretrained_word_vectors/ft_cbow_200d_20e.vec"
wvs = global_utils.get_corpus_wvs_from_ft(fasttext_vec_file, 200, vocab_words)
wvs.shape
Explanation: FastText Vectors
fasttext commands used
fasttext skipgram -minCount 1 -dim 200 -epoch 10 -input corpus_text_for_fast_text.txt -output ft_wvs_200d_10e
fasttext cbow -minCount 1 -dim 200 -epoch 10 -input corpus_text_for_fast_text.txt -output ft_wvs_200d_10e
reading ft vectors
End of explanation
%ll /home/bicepjai/Projects/dsotc/data_prep/processed/stage1/pretrained_word_vectors
len(vocab_words)
%autoreload
import global_utils
ft_vector_files = [
                   (100,"ft_cbow_100d_20e"),(200,"ft_cbow_200d_20e"),(200,"ft_cbow_300d_20e"),
                   (100,"ft_sg_100d_20e"),(200,"ft_sg_200d_20e"),(200,"ft_sg_300d_20e"),
                   (100,"ft_cbow_100d_50e"),(200,"ft_cbow_200d_50e"),(200,"ft_cbow_300d_50e"),
                   (100,"ft_sg_100d_50e"),(200,"ft_sg_200d_50e"),(200,"ft_sg_300d_50e"),
                   (100,"ft_cbow_100d_100e"),(200,"ft_cbow_200d_100e"),(200,"ft_cbow_300d_100e"),
                   (100,"ft_sg_100d_100e"),(200,"ft_sg_200d_100e"),(200,"ft_sg_300d_100e")
                  ]
for dim_file_name in ft_vector_files:
    file_path = "/home/bicepjai/Projects/dsotc/data_prep/processed/stage1/pretrained_word_vectors/"+dim_file_name[1]+".vec"
    dim = dim_file_name[0]
    if not os.path.exists(file_path):
        print("file doesnt exist",file_path)
        continue
    ft_vec = global_utils.get_corpus_wvs_from_ft(file_path, dim, vocab_words)
    print(ft_vector_file,ft_vec.shape)
    np.save("processed/stage1/pretrained_word_vectors/"+dim_file_name[1]+".npy", ft_vec)
Explanation: saving all trained fast text vectors
End of explanation
%autoreload
import global_utils
WORD_EMB_SIZE=200
ft_file_path = "/home/bicepjai/Projects/Deep-Survey-Text-Classification/data_prep/processed/stage1/pretrained_word_vectors/ft_sg_200d_50e.vec"
trained_embeddings = global_utils.get_embeddings_from_ft(ft_file_path, WORD_EMB_SIZE, corpus_vocab_list)
trained_embeddings.shape
tb_vocab_size=5000
tb_vocab_biolab = list(trained_embeddings)[:tb_vocab_size]
with open("view_wvs_tb/tb_vocab.tsv", "w") as fp:
    wr = csv.writer(fp, delimiter='\n')
    wr.writerow(corpus_vocab_list)
tb_word_vectors = np.random.randn(tb_vocab_size, 200)
for i,word in enumerate(tb_vocab_biolab):
    tb_word_vectors[i] = trained_embeddings[i]
%autoreload
from utils import visualize_embeddings_in_tensorboard
visualize_this_embedding = tb_word_vectors
print(visualize_this_embedding.shape)
metadata_path = "/home/bicepjai/Projects/Deep-Survey-Text-Classification/data_prep/view_wvs_tb/tb_vocab.tsv"
visualize_embeddings_in_tensorboard(visualize_this_embedding, metadata_path, "/home/bicepjai/Projects/Deep-Survey-Text-Classification/data_prep/view_wvs_tb")
Explanation: Viewing word vectors
End of explanation
<END_TASK>