<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 corpushash tutorial
corpushash is a simple library that aims to make the natural language processing of sensitive documents easier. the library enables performing common NLP tasks on sensitive documents without disclosing their contents. This is done by hashing every token in the corpus along with a salt (to prevent dictionary attacks). 
its workflow is as simple as having the sensitive corpora as a python nested list (or generator) whose elements are themselves (nested) lists of strings. after the hashing is done, NLP can be carried out by a third party, and when the results are in they can be decoded by a dictionary that maps hashes to the original strings. so that makes
Step1:  we'll use the gutenberg corpus as test data, which is available through the nltk library.
downloading test data (if needed)
Step2:  files in test data
Step3:  creating test corpus path, where hashed documents will be stored as .json files
Step4:  function to split text into nested list
Step5:  every paragraph and sentence is its own list
Step6:  Input
the library takes as input a nested list whose elements are the original documents as nested lists. this can be an in-nested list or some generator that yields a nested list when it is iterated over.
creating nested list made from the raw texts in the gutenberg corpus
Step7:  excerpt
Step8:  processing using corpushash
instantiating CorpusHash class, which hashes the provided corpus to the corpus_path
Step9:  Output
Encode dictionary
The encode dictionary is used to encode values to hashes, so that the same strings are guaranteed to be hashed to the same value, including the random salt.
Step10:  Decode dictionary
The decode dictionary is used to decode hashes to their original strings, so that one can make sense of the results of any posterior NLP analysis. It also lists the salt that was used to obtain the given hash. This dictionary must be kept secret with the owner of the data.
Step11:  hashed .json files
the walk_nested_list function yields items in a nested list in order, regardless of their depth
Step12:  we can use this function to see what hashcodecs has done to the corpus.
adjusting parameters
Step13:  note
Step14:  or using corpushash's read_hashed_corpus method, which yields the corpus' documents in order
Step15:  alternatively, one can check the corpus_path directory and read the output files using one's favorite text editor.
the path is
<ASSISTANT_TASK:>
Python Code:
import os
import json
from nltk.corpus import gutenberg
import corpushash as ch
import base64
import hashlib
import random
Explanation: corpushash tutorial
corpushash is a simple library that aims to make the natural language processing of sensitive documents easier. the library enables performing common NLP tasks on sensitive documents without disclosing their contents. This is done by hashing every token in the corpus along with a salt (to prevent dictionary attacks). 
its workflow is as simple as having the sensitive corpora as a python nested list (or generator) whose elements are themselves (nested) lists of strings. after the hashing is done, NLP can be carried out by a third party, and when the results are in they can be decoded by a dictionary that maps hashes to the original strings. so that makes:
```python
import corpushash as ch
hashed_corpus = ch.CorpusHash(mycorpus_as_a_nested_list, '/home/sensitive-corpus')
"42 documents hashed and saved to '/home/sensitive-corpus/public/$(timestamp)'"
**NLP is done, and `results` are in**:python
for token in results:
    print(token, ">", hashed_corpus.decode_dictionary[token])
"7)JBMGG?sGu+>%Js~dG=%c1Qn1HpAU{jM-~Buu7?" > "gutenberg"
```
The library requires as input:
a tokenized corpus as a nested list, whose elements are themselves nested lists of the tokens of each document in the corpus
each list corresponds to a document structure: its chapters, paragraphs, sentences. you decide how the nested list is to be created or structured, as long as the input is a nested list with strings as their bottom-most elements.
corpus_path, a path to a directory where the output files are to be stored
The output includes:
a .json file for every item in the dictionary provided, named sequencially as positive integers, e.g., the first document being 0.json, stored in corpus_path/public/$(timestamp-of-hash)/
two .json dictionaries stored in corpus_path/private. they are used to decode the .json files or the NLP results
Preparing the input data
loading libraries...
End of explanation
import nltk
#nltk.download('gutenberg')  # comment (uncomment) if you have (don't have) the data
Explanation: we'll use the gutenberg corpus as test data, which is available through the nltk library.
downloading test data (if needed):
End of explanation
gutenberg.fileids()
Explanation: files in test data:
End of explanation
base_path = os.getcwd()
base_path
corpus_path = os.path.join(base_path, 'guten_test')
corpus_path
Explanation: creating test corpus path, where hashed documents will be stored as .json files:
End of explanation
excerpt = gutenberg.raw('austen-emma.txt')[50:478]
print(excerpt)
Explanation: function to split text into nested list:
End of explanation
print(ch.text_split(excerpt))
Explanation: every paragraph and sentence is its own list:
End of explanation
%%time
guten_list = []
for document_name in gutenberg.fileids():
    document = gutenberg.raw(document_name)
    split_document = ch.text_split(document)
    guten_list.append(split_document)
Explanation: Input
the library takes as input a nested list whose elements are the original documents as nested lists. this can be an in-nested list or some generator that yields a nested list when it is iterated over.
creating nested list made from the raw texts in the gutenberg corpus:
End of explanation
document = random.choice(guten_list)
print(document[:10])
Explanation: excerpt:
End of explanation
%time hashed_guten = ch.CorpusHash(guten_list, corpus_path)
Explanation: processing using corpushash
instantiating CorpusHash class, which hashes the provided corpus to the corpus_path:
End of explanation
entries = random.sample(list(hashed_guten.encode_dictionary.keys()), k=5)
for entry in entries:
    print("token >> {:^20} | hashed_token >> '{}'".format(entry, hashed_guten.encode_dictionary[entry]))
Explanation: Output
Encode dictionary
The encode dictionary is used to encode values to hashes, so that the same strings are guaranteed to be hashed to the same value, including the random salt.
End of explanation
entries = random.sample(list(hashed_guten.decode_dictionary.keys()), k=5)
for entry in entries:
    print("hashed_token >> '{}' | (token >> '{}', salt >> '{}'".format(entry, hashed_guten.decode_dictionary[entry][0], hashed_guten.decode_dictionary[entry][1][:4]))  # cutting off some bytes for aesthetic reasons
Explanation: Decode dictionary
The decode dictionary is used to decode hashes to their original strings, so that one can make sense of the results of any posterior NLP analysis. It also lists the salt that was used to obtain the given hash. This dictionary must be kept secret with the owner of the data.
End of explanation
print(excerpt)
for element in ch.walk_nested_list(ch.text_split(excerpt)):
    print(element)
Explanation: hashed .json files
the walk_nested_list function yields items in a nested list in order, regardless of their depth:
End of explanation
limit = 10  # showing first ten entries
document = random.randint(0, len(gutenberg.fileids()))
print('document {} corresponds to {}.'.format(document, gutenberg.fileids()[document]))
Explanation: we can use this function to see what hashcodecs has done to the corpus.
adjusting parameters:
End of explanation
document_path = os.path.join(hashed_guten.public_path, "{}.json".format(document))
with open(document_path, mode="rt") as fp:
    encoded_document = json.load(fp)
print("original token >> encoded token")
for ix, tokens in enumerate(zip(ch.walk_nested_list(encoded_document), ch.walk_nested_list(encoded_document))):
    print("'{}' >> '{}'".format(tokens[0], tokens[1]))
    if ix > limit:
        break
Explanation: note: take care to have your corpus yield the documents always in the same order, else you'll have a harder time knowing which hashed documents correspond to which documents.
you can read output .json directly:
End of explanation
for document in hashed_guten.read_hashed_corpus():
    print(document[0])
    break
Explanation: or using corpushash's read_hashed_corpus method, which yields the corpus' documents in order:
End of explanation
hashed_guten.public_path
Explanation: alternatively, one can check the corpus_path directory and read the output files using one's favorite text editor.
the path is:
End of explanation
<END_TASK>