<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 doc2vec
This is an experimental code developed by Tomas Mikolov found and published on the word2vec Google group.
The input format for doc2vec is still one big text document but every line should be one document prepended with an unique id, for example
Step1:  doc2vec
Step2:  Prediction
Is possible to load the vectors using the same wordvectors class as a regular word2vec binary file.
Step3:  The documents vector are going to be identified by the id we used in the preprocesing section, for example document 1 is going to have vector
Step4:  We can ask for similarity words or documents on document 1
<ASSISTANT_TASK:>
Python Code:
import os
import nltk
directories = ['train/pos', 'train/neg', 'test/pos', 'test/neg', 'train/unsup']
input_file = open('../data/alldata.txt', 'w')
id_ = 0
for directory in directories:
    rootdir = os.path.join('../data/aclImdb', directory)
    for subdir, dirs, files in os.walk(rootdir):
        for file_ in files:
            with open(os.path.join(subdir, file_), "r") as f:
                doc_id = "_*%i" % id_
                id_ = id_ + 1
                text = f.read()
                text = text
                tokens = nltk.word_tokenize(text)
                doc = " ".join(tokens).lower()
                doc = doc.encode("ascii", "ignore")
                input_file.write("%s %s\n" % (doc_id, doc))
input_file.close()
Explanation: doc2vec
This is an experimental code developed by Tomas Mikolov found and published on the word2vec Google group.
The input format for doc2vec is still one big text document but every line should be one document prepended with an unique id, for example:
_*0 This is sentence 1
_*1 This is sentence 2
Requirements
This notebook requires nltk
Download some data: http://ai.stanford.edu/~amaas/data/sentiment/aclImdb_v1.tar.gz.
You could use make test-data from the root of the repo.
Preprocess
Merge data into one big document with an id per line and do some basic preprocessing: word tokenizer.
End of explanation
%load_ext autoreload
%autoreload 2
import word2vec
word2vec.doc2vec('../data/alldata.txt', '../data/doc2vec-vectors.bin', cbow=0, size=100, window=10, negative=5,
                 hs=0, sample='1e-4', threads=12, iter_=20, min_count=1, binary=True, verbose=True)
Explanation: doc2vec
End of explanation
%load_ext autoreload
%autoreload 2
import word2vec
model = word2vec.load('../data/doc2vec-vectors.bin')
model.vectors.shape
Explanation: Prediction
Is possible to load the vectors using the same wordvectors class as a regular word2vec binary file.
End of explanation
model['_*1']
Explanation: The documents vector are going to be identified by the id we used in the preprocesing section, for example document 1 is going to have vector:
End of explanation
indexes, metrics = model.similar('_*1')
model.generate_response(indexes, metrics).tolist()
Explanation: We can ask for similarity words or documents on document 1
End of explanation
<END_TASK>