R text2vec and rword2vec analogy results differ between packages
<p>When I use the R rword2vec package to calculate word analogies on a body of text, I get great, intuitive answers. When I output the word vectors from this using bin_to_txt, however, and then read these in and calculate the analogy using text2vec package, I get really low-quality answers.</p>

<p>Is there a difference in how the two packages calculate the analogies, and if so, how can I reproduce rword2vec's calculation without this package? Unfortunately I can't install rword2vec on the machine I need to use in production.</p>

<p>In the example below, I use the analogy king:queen::man:_. With the same vectors, rword2vec gives me "woman", "girl", and "blonde" and text2vec method using sim2 gives me "man", "ahab", and "king". What am I messing up with the latter methodology?</p>

<p>Code below for reproducing:</p>

<pre><code># Text file: http://mattmahoney.net/dc/text8.zip
# USING rword2vec
# require(devtools)
# install_github("mukul13/rword2vec")
require(rword2vec)

file1="C:\\Users\\bdk\\Downloads\\text8"

model = word2vec(train_file = file1, output_file = "vec_repro.bin",binary=1)

anal1 = word_analogy(file_name = "vec_repro.bin",search_words = "king queen man",num = 20)
print(anal1)
# first 10 results:
# word              dist
# 1       woman 0.718326687812805
# 2        girl 0.607264637947083
# 3      blonde  0.56832879781723
# 4        lady 0.518971383571625
# 5      lovely 0.515585899353027
# 6    stranger 0.504840195178986
# 7      ladies 0.500177025794983
# 8      totoro 0.497228592634201
# 9        baby 0.497049778699875
# 10   handsome 0.490864992141724

bin_to_txt("vec_repro.bin","vector_repro.txt")

# Read in these word vectors and do the same calculation but with text2vec
require(text2vec)
data1=as.data.frame(read.table("vector_repro.txt",skip=1))
vocab_all2 &lt;- data1
rownames(vocab_all2) &lt;- vocab_all2[,1]
vocab_all2$V1 &lt;- NULL
colnames(vocab_all2) &lt;- NULL
vocab_all2 &lt;- vocab_all2[complete.cases(vocab_all2),]
vocab_all3 &lt;- data.matrix(vocab_all2)

guess1 &lt;- (
  vocab_all3["king", , drop = FALSE] -
    vocab_all3["queen", , drop = FALSE] +
    vocab_all3["man", , drop = FALSE]
)

cos_sim = sim2(x = vocab_all3, y = guess1, method = "cosine", norm = "l2")
head(sort(cos_sim[,1], decreasing = TRUE), 10)
# first 10 results:
# man      ahab      king       god   prophet   faramir   saladin      saul      enki   usurper 
# 0.7212826 0.4715135 0.4696279 0.4625656 0.4522798 0.4391127 0.4358722 0.4326022 0.4310836 0.4300992 
</code></pre>