<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 TF-IDF on gutenberg corpus
corpushash is a simple library that aims to make the natural language processing of sensitive documents easier. the library enables performing common NLP tasks on sensitive documents without disclosing their contents. This is done by hashing every token in the corpus along with a salt (to prevent dictionary attacks). 
its workflow is as simple as having the sensitive corpora as a python nested list (or generator) whose elements are themselves (nested) lists of strings. after the hashing is done, NLP can be carried out by a third party, and when the results are in they can be decoded by a dictionary that maps hashes to the original strings. so that makes
Step1:  downloading nltk gutenberg corpus, if not downloaded already
Step2:  files in test data
Step3:  preparing input
creating test corpus path, where hashed documents will be stored as .json files
Step4:  the library takes as input a nested list whose elements are the original documents as nested lists. the nested lists represent the document's structure (sections, paragraphs, sentences). this can be an in-memory nested list or some generator that yields a nested list when it is iterated over.
as this is a simple tf-idf transformation, we don't need the nested list format, because document structure is not important
Step5:  excerpt
Step6:  processing using corpushash
loading libraries for corpushash
Step7:  instatiating CorpusHash class, which hashes the provided corpus to the corpus_path
Step8:  that is it. corpushash's work is done.
NLP
Step9:  loading libraries we need for processing
Step10:  defining iterable for gensim
Step11:  as the analyst will not have access to this convenience function, we will build a new generator for this.
Step12:  building gensim dictionary
from this document generator gensim will build a dictionary that maps every hashed token to an ID, a mapping which is later used to calculate the tf-idf weights
Step13:  bag-of-words
to build a tf-idf model, the gensim library needs an input that yields this vectorized bag-of-words when iterated over
Step14:  we must re-instantiate the generator, else it'll be depleted.
Step15:  tf-idf model
now we are ready to deploy the tf-idf model to our corpus.
when applying gensim.models.TfidfModel we calculate the tf-idf weights of every token in the corpus.
Step16:  after calculating the frequencies, its time to transform the given bag-of-words vectors to corresponding tf-idf weights vectors.
Step17:  example of token ids and their tf-idf weights
Step18:  validating tf-idf model on unencoded corpus
now we will take the role of the corpus owner when she gets the results from the analyst back.
we will validate the results in two steps
Step19:  obtaining decode dictionary
Step20:  here we are reinstantiating the generators, applying the tf-idf model, and then iterating over the model's results to replace the hashed tokens with their decoded counterparts. the result is saved to disk. make sure your indexer doesn't change document order (gutenberg.fileids does it), else the names may mismatch with the contents.
Step21:  2. tf-idf on decoded corpus
creating unencoded corpus dictionary
the next step in our validation procedure is to apply the same analysis to the the decoded corpus.
the input files are not in the /public/$(timestamp) directory, but in the decoded_gutencorpus variable.
the files are, again, decoded, in contrast to the hashed tokens seen above.
Step22:  creating the dictionary that maps a token to an ID
Step23:  creating a generator that yields the bag-of-words model of a document when iterated over
Step24:  tf-idf model
creating the model
Step25:  applying the model to the documents (not forgetting to reinstantiate the generators)
Step26:  comparing results of encoded and decoded corpus
reinstatiating the generators...
<ASSISTANT_TASK:>
Python Code:
import os
import string
import random
import pickle
import nltk
from nltk.corpus import gutenberg
Explanation: TF-IDF on gutenberg corpus
corpushash is a simple library that aims to make the natural language processing of sensitive documents easier. the library enables performing common NLP tasks on sensitive documents without disclosing their contents. This is done by hashing every token in the corpus along with a salt (to prevent dictionary attacks). 
its workflow is as simple as having the sensitive corpora as a python nested list (or generator) whose elements are themselves (nested) lists of strings. after the hashing is done, NLP can be carried out by a third party, and when the results are in they can be decoded by a dictionary that maps hashes to the original strings. so that makes:
```python
import corpushash as ch
hashed_corpus = ch.CorpusHash(mycorpus_as_a_nested_list, '/home/sensitive-corpus')
"42 documents hashed and saved to '/home/sensitive-corpus/public/$(timestamp)'"
**NLP is done, and `results` are in**:python
for token in results:
    print(token, ">", hashed_corpus.decode_dictionary[token])
"7)JBMGG?sGu+>%Js~dG=%c1Qn1HpAU{jM-~Buu7?" > "gutenberg"
```
loading libraries for preprocessing
End of explanation
#nltk.download("gutenberg")
Explanation: downloading nltk gutenberg corpus, if not downloaded already:
End of explanation
gutenberg.fileids()
Explanation: files in test data:
End of explanation
corpus_path = os.path.join(os.getcwd(), 'guten_test')
corpus_path
Explanation: preparing input
creating test corpus path, where hashed documents will be stored as .json files:
End of explanation
%%time
decoded_gutencorpus = []
for document_name in gutenberg.fileids():
    document = [word.lower() for word in gutenberg.words(document_name) if word not in string.punctuation and not word.isdigit()]
    decoded_gutencorpus.append(document)
Explanation: the library takes as input a nested list whose elements are the original documents as nested lists. the nested lists represent the document's structure (sections, paragraphs, sentences). this can be an in-memory nested list or some generator that yields a nested list when it is iterated over.
as this is a simple tf-idf transformation, we don't need the nested list format, because document structure is not important: just having the words with no sentence or paragraph structure is sufficient. the library's input is designed to be flexible in this regard.
creating a dictionary whose keys are the fileids and whose values are a list of the words in every document:
End of explanation
document = random.choice(decoded_gutencorpus)
print(document[:100])
Explanation: excerpt:
End of explanation
import corpushash as ch
Explanation: processing using corpushash
loading libraries for corpushash
End of explanation
%time hashed_guten = ch.CorpusHash(decoded_gutencorpus, corpus_path)
Explanation: instatiating CorpusHash class, which hashes the provided corpus to the corpus_path:
End of explanation
encoded_corpus_path = hashed_guten.public_path
encoded_corpus_path
os.path.exists(encoded_corpus_path)
Explanation: that is it. corpushash's work is done.
NLP: a tf-idf example
from now on we are simulating the work of an analyst that receives the encoded documents and performs a common NLP task on them, the calculation of tf-idf weights.
note: we will be using the gensim library for the NLP. if you are not familiar with it, you may want to check the first couple tutorials. (you could use the library of your choice, naturally).
all the analyst has are the files on corpushash/gutenberg_test/public/$(timestamp).
End of explanation
import json
import gensim
Explanation: loading libraries we need for processing
End of explanation
for i in hashed_guten.read_hashed_corpus():
    print(i[0], i[1][:1])
Explanation: defining iterable for gensim:
in order to perform the tf-idf calculation, we use the gensim library. the gensim library takes as input any object that yields documents when it is iterated over.
corpushash has a built-in reader that yields the document index and the hashed document. these are the first tokens in each document:
End of explanation
def encoded_gutenberg_yielder(corpus_path):
    for ix in range(len(gutenberg.fileids())):
        path = os.path.join(corpus_path, '{}.json'.format(ix))
        with open(path, 'r') as fp:
            document_tokens = json.load(fp)
        yield document_tokens
example_doc = encoded_gutenberg_yielder(encoded_corpus_path)
print("\n".join(next(example_doc)[:10]))
Explanation: as the analyst will not have access to this convenience function, we will build a new generator for this.
End of explanation
encoded_gutencorpus = encoded_gutenberg_yielder(encoded_corpus_path)
encoded_gutendict = gensim.corpora.Dictionary(encoded_gutencorpus)
#encoded_gutendict.save_as_text('enc_dict.txt', sort_by_word=False)
print("the number of unique words in our corpus is {}.".format(len(encoded_gutendict)))
Explanation: building gensim dictionary
from this document generator gensim will build a dictionary that maps every hashed token to an ID, a mapping which is later used to calculate the tf-idf weights:
End of explanation
def bow_gutenberg_yielder(corpus, dictionary):
    for document_tokens in corpus:
        yield dictionary.doc2bow(document_tokens)
Explanation: bag-of-words
to build a tf-idf model, the gensim library needs an input that yields this vectorized bag-of-words when iterated over:
End of explanation
encoded_gutencorpus = encoded_gutenberg_yielder(encoded_corpus_path)
encoded_gutenbow = bow_gutenberg_yielder(encoded_gutencorpus, encoded_gutendict)
print('token', '>>', "(token id, frequency in document)\n")
for i in next(encoded_gutenbow)[:10]:
    print(encoded_gutendict.get(i[0]), '>>', i)
Explanation: we must re-instantiate the generator, else it'll be depleted.
End of explanation
encoded_gutencorpus = encoded_gutenberg_yielder(encoded_corpus_path)
encoded_gutenbow = bow_gutenberg_yielder(encoded_gutencorpus, encoded_gutendict)
encoded_tfidf = gensim.models.TfidfModel(encoded_gutenbow)
Explanation: tf-idf model
now we are ready to deploy the tf-idf model to our corpus.
when applying gensim.models.TfidfModel we calculate the tf-idf weights of every token in the corpus.
End of explanation
encoded_gutencorpus = encoded_gutenberg_yielder(encoded_corpus_path)
encoded_gutenbow = bow_gutenberg_yielder(encoded_gutencorpus, encoded_gutendict)
encoded_guten_tfidf = encoded_tfidf[encoded_gutenbow]
Explanation: after calculating the frequencies, its time to transform the given bag-of-words vectors to corresponding tf-idf weights vectors.
End of explanation
print('token', '>>', "(token id, tf-idf weight)\n")
for i in encoded_guten_tfidf:
    for k in i[:10]:
        print(encoded_gutendict.get(k[0]), '>>', k)
    break
Explanation: example of token ids and their tf-idf weights:
End of explanation
decode_dictionary_path = hashed_guten.decode_dictionary_path
os.path.isfile(decode_dictionary_path)
Explanation: validating tf-idf model on unencoded corpus
now we will take the role of the corpus owner when she gets the results from the analyst back.
we will validate the results in two steps:
we will decode the previous result back to the unhashed tokens, maintaining their tf-idf weights;
we will compare the tf-idf results with the same analysis done on the decoded corpus.
1. decoding previous result
we will turn the previous result from a tuple as in (6, 0.0017337194574225342) to something like genesis: 0.0017337194574225342 and store it in disk.
End of explanation
with open(decode_dictionary_path, 'rb') as f:
    decode_dictionary = pickle.load(f)
Explanation: obtaining decode dictionary:
End of explanation
encoded_gutencorpus = encoded_gutenberg_yielder(encoded_corpus_path)
encoded_gutenbow = bow_gutenberg_yielder(encoded_gutencorpus, encoded_gutendict)
encoded_guten_tfidf = encoded_tfidf[encoded_gutenbow]
for path, document in zip(gutenberg.fileids(), encoded_guten_tfidf):
    decoded_document = []
    for tuple_value in document:
        hashed_token = encoded_gutendict.get(tuple_value[0]) # 6 - > ed07dbbe94c8ff385a1a00e6720f0ab66ac420...
        token, _ = decode_dictionary[hashed_token]  # 'ed07dbbe94c8ff385a1a00e... -> 'genesis'
        decoded_document.append("{}: {}".format(token, tuple_value[1]))
    fname = 'decoded_'+ path
    with open(os.path.join(corpus_path, fname), 'w') as f:
        f.write("\n".join(decoded_document))
example_id = random.choice(gutenberg.fileids())
with open(os.path.join(corpus_path, 'decoded_' + example_id), 'r') as f:
        decoded_doc = f.read().splitlines()
print(example_id, '>>')
print("\n".join(decoded_doc[:10]))
Explanation: here we are reinstantiating the generators, applying the tf-idf model, and then iterating over the model's results to replace the hashed tokens with their decoded counterparts. the result is saved to disk. make sure your indexer doesn't change document order (gutenberg.fileids does it), else the names may mismatch with the contents.
End of explanation
example_document = random.choice(decoded_gutencorpus)
print(example_document[:10])
Explanation: 2. tf-idf on decoded corpus
creating unencoded corpus dictionary
the next step in our validation procedure is to apply the same analysis to the the decoded corpus.
the input files are not in the /public/$(timestamp) directory, but in the decoded_gutencorpus variable.
the files are, again, decoded, in contrast to the hashed tokens seen above.
End of explanation
decoded_gutendict = gensim.corpora.Dictionary(decoded_gutencorpus)
#decoded_gutendict.save_as_text('dec_dict.txt', sort_by_word=False)
len(decoded_gutendict)
Explanation: creating the dictionary that maps a token to an ID:
End of explanation
decoded_gutenbow = bow_gutenberg_yielder(decoded_gutencorpus, decoded_gutendict)
print('token', '>>', "(token id, frequency in document)\n")
for i in next(decoded_gutenbow)[:10]:
    print(decoded_gutendict.get(i[0]), '>>', i)
Explanation: creating a generator that yields the bag-of-words model of a document when iterated over:
End of explanation
decoded_gutenbow = bow_gutenberg_yielder(decoded_gutencorpus, decoded_gutendict)
decoded_tfidf = gensim.models.TfidfModel(decoded_gutenbow)
Explanation: tf-idf model
creating the model:
End of explanation
decoded_gutenbow = bow_gutenberg_yielder(decoded_gutencorpus, decoded_gutendict)
decoded_guten_tfidf = decoded_tfidf[decoded_gutenbow]
print('token', '>>', "(token id, tf-idf weight)\n")
for i in decoded_guten_tfidf:
    for k in i[:10]:
        print(decoded_gutendict.get(k[0]), '>>', k)
    break
Explanation: applying the model to the documents (not forgetting to reinstantiate the generators):
End of explanation
encoded_gutencorpus = encoded_gutenberg_yielder(encoded_corpus_path)
encoded_gutenbow = bow_gutenberg_yielder(encoded_gutencorpus, encoded_gutendict)
encoded_guten_tfidf = encoded_tfidf[encoded_gutenbow]
decoded_gutenbow = bow_gutenberg_yielder(decoded_gutencorpus, decoded_gutendict)
decoded_guten_tfidf = decoded_tfidf[decoded_gutenbow]
%%time
encoded_tfidf, decoded_tfidf = {}, {}
for encoded_document, decoded_document in zip(encoded_guten_tfidf, decoded_guten_tfidf):
    for encoded_item, decoded_item in zip(encoded_document, decoded_document):
        hashed_token = encoded_gutendict.get(encoded_item[0])
        original_token = decode_dictionary[hashed_token][0] # get hash, ignoring salt
        encoded_tfidf[original_token] = round(encoded_item[1], 7) # rounding because python <3.6 seems to represent floats inconsistently
        decoded_tfidf[decoded_gutendict.get(decoded_item[0])] = round(decoded_item[1], 7)
print(encoded_tfidf == decoded_tfidf)
random_token = random.choice(list(encoded_tfidf.keys()))
print("example token: tf-idf weight in encoded corpus | in decoded corpus\n{:^35}: {} | {}".format(random_token, encoded_tfidf[random_token], decoded_tfidf[random_token]))
Explanation: comparing results of encoded and decoded corpus
reinstatiating the generators...
End of explanation
<END_TASK>