<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 Example adapted from
Step1:  Load IMDB Dataset
Step2:  Pad sequences so they are all the same length (required by keras/tensorflow).
Step3:  Setup Vocabulary Dictionary
The index value loaded differes from the dictionary value by "index_from" so that special characters for padding, start of sentence, and out of vocabulary can be prepended to the start of the vocabulary.
Step4:  Convert Encoded Sentences to Readable Text
Step5:  Build the model
Sequential guide, compile() and fit() 
Embedding The embeddings layer works like an effiecient one hot encoding for the word index followed by a dense layer of size embedding_vector_length.
LSTM (middle of page)
Dense
Dropout (1/3 down the page)
"model.compile(...) sets up the "adam" optimizer, similar to SGD but with some gradient averaging that works like a larger batch size to reduce the variability in the gradient from one small batch to the next.  Each SGD step is of batch_size training records. Adam is also a variant of momentum optimizers.
'binary_crossentropy' is the loss functiom used most often with logistic regression and is equivalent to softmax for only two classes.
In the "Output Shape", None is a unknown for a variable number of training records to be supplied later.
Step6:  Setup Tensorboard
make sure /data/kaggle-tensorboard path exists or can be created
Start tensorboard from command line
Step7:  Train the Model
Each epoch takes about 3 min. You can reduce the epochs to 3 for a faster build and still get good accuracy. Overfitting starts to happen at epoch 7 to 9.
Note
Step8:  Accuracy on the Test Set
Step9:  # Hyper Parameter Tuning Notes
| Accuracy %    | Type      | max val acc epoch | embedding_vector_length  | RNN state size |  Dropout
| 
Step10:  Evaluate on Custom Text
Step11:  Features View
Step12:  Results
<ASSISTANT_TASK:>
Python Code:
import numpy as np
from keras.datasets import imdb
from keras.models import Sequential
from keras.layers import Dense, LSTM, GRU, Dropout
from keras.layers.embeddings import Embedding
from keras.preprocessing import sequence
from keras.callbacks import TensorBoard
from keras import backend 
# fix random seed for reproducibility
np.random.seed(7)
import shutil
import os
import matplotlib.pyplot as plt
%matplotlib inline
from IPython.core.display import display, HTML
display(HTML("<style>.container { width:97% !important; }</style>")) #Set width of iPython cells
Explanation: Example adapted from: https://machinelearningmastery.com/sequence-classification-lstm-recurrent-neural-networks-python-keras/
Required instals:
1. jupyter notebooks
2. keras 
3. tensorflow
My install process was:
1. follow instructions for python virtual environment (Virtualenv) install at https://www.tensorflow.org/install/
2. install keras python env using https://keras.io/#installation
3. install jupyter notebooks (http://jupyter.org/install) and set up a tensorflow kernel that uses the virtualenv set up above. 
4. start jupyter notebooks in a parent directory of this notebook and open this notebook. Make sure the Tensorflow Virtualenv jupyter kernel is active when running the notebook.
The logs of my install are at: https://www.evernote.com/l/ACtXalW9qSpOVZOUU04V2ATOmJOvw4Ffido
End of explanation
# load the dataset but only keep the top n words, zero the rest
# docs at: https://www.tensorflow.org/api_docs/python/tf/keras/datasets/imdb/load_data 
top_words = 5000
start_char=1
oov_char=2
index_from=3
(X_train, y_train), (X_test, y_test) = imdb.load_data(num_words=top_words, 
                                start_char=start_char, oov_char = oov_char, index_from = index_from )
print(X_train.shape)
print(y_train.shape)
print(len(X_train[0]))
print(len(X_train[1]))
print(X_test.shape)
print(y_test.shape)
X_train[0]
Explanation: Load IMDB Dataset
End of explanation
# truncate and pad input sequences
max_review_length = 500
X_train = sequence.pad_sequences(X_train, maxlen=max_review_length)
X_test = sequence.pad_sequences(X_test, maxlen=max_review_length)
print(X_train.shape)
print(y_train.shape)
print(len(X_train[0]))
print(len(X_train[1]))
print(X_test.shape)
print(y_test.shape)
X_train[0]
y_train[0:20]  # first 20 sentiment labels
Explanation: Pad sequences so they are all the same length (required by keras/tensorflow).
End of explanation
word_index = imdb.get_word_index()
inv_word_index = np.empty(len(word_index)+index_from+3, dtype=np.object)
for k, v in word_index.items():
    inv_word_index[v+index_from]=k
inv_word_index[0]='<pad>'    
inv_word_index[1]='<start>'
inv_word_index[2]='<oov>' 
word_index['ai']
inv_word_index[16942+index_from]
inv_word_index[:50]
Explanation: Setup Vocabulary Dictionary
The index value loaded differes from the dictionary value by "index_from" so that special characters for padding, start of sentence, and out of vocabulary can be prepended to the start of the vocabulary.
End of explanation
def toText(wordIDs):
    s = ''
    for i in range(len(wordIDs)):
        if wordIDs[i] != 0:
            w = str(inv_word_index[wordIDs[i]])
            s+= w + ' '
    return s
for i in range(5):
    print()
    print(str(i) + ') sentiment = ' + ('negative' if y_train[i]==0 else 'positive'))
    print(toText(X_train[i]))
Explanation: Convert Encoded Sentences to Readable Text
End of explanation
backend.clear_session()
embedding_vector_length = 5
rnn_vector_length = 150
#activation = 'relu'
activation = 'sigmoid'
    
model = Sequential()
model.add(Embedding(top_words, embedding_vector_length, input_length=max_review_length))
model.add(Dropout(0.2))
#model.add(LSTM(rnn_vector_length, activation=activation))
model.add(GRU(rnn_vector_length, activation=activation))
model.add(Dropout(0.2))
model.add(Dense(1, activation=activation))
model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy'])
print(model.summary())
Explanation: Build the model
Sequential guide, compile() and fit() 
Embedding The embeddings layer works like an effiecient one hot encoding for the word index followed by a dense layer of size embedding_vector_length.
LSTM (middle of page)
Dense
Dropout (1/3 down the page)
"model.compile(...) sets up the "adam" optimizer, similar to SGD but with some gradient averaging that works like a larger batch size to reduce the variability in the gradient from one small batch to the next.  Each SGD step is of batch_size training records. Adam is also a variant of momentum optimizers.
'binary_crossentropy' is the loss functiom used most often with logistic regression and is equivalent to softmax for only two classes.
In the "Output Shape", None is a unknown for a variable number of training records to be supplied later.
End of explanation
log_dir = '/data/kaggle-tensorboard'
shutil.rmtree(log_dir, ignore_errors=True)
os.makedirs(log_dir)
tbCallBack = TensorBoard(log_dir=log_dir, histogram_freq=0, write_graph=True, write_images=True)
full_history=[]
Explanation: Setup Tensorboard
make sure /data/kaggle-tensorboard path exists or can be created
Start tensorboard from command line:
tensorboard --logdir=/data/kaggle-tensorboard
open http://localhost:6006/
End of explanation
history = model.fit(X_train, y_train, validation_data=(X_test, y_test), epochs=8, batch_size=64, callbacks=[tbCallBack])
full_history += history.history['loss']
Explanation: Train the Model
Each epoch takes about 3 min. You can reduce the epochs to 3 for a faster build and still get good accuracy. Overfitting starts to happen at epoch 7 to 9.
Note: You can run this cell multiple times to add more epochs to the model training without starting over.
End of explanation
scores = model.evaluate(X_test, y_test, verbose=0)
print("Accuracy: %.2f%%" % (scores[1]*100))
print( 'embedding_vector_length = ' + str( embedding_vector_length ))
print( 'rnn_vector_length = ' + str( rnn_vector_length ))
Explanation: Accuracy on the Test Set
End of explanation
history.history  # todo: add graph of all 4 values with history
plt.plot(history.history['loss'])
plt.yscale('log')
plt.show()
plt.plot(full_history)
plt.yscale('log')
plt.show()
Explanation: # Hyper Parameter Tuning Notes
| Accuracy %    | Type      | max val acc epoch | embedding_vector_length  | RNN state size |  Dropout
| :------------ | :-------- | :----  | :----                    |  :----         |  :----
| 88.46     *   | GRU       | 6      | 5                        | 150            | 0.2 (after Embedding and LSTM)
| 88.4          | GRU       | 4      | 5                        | 100            | no dropout
| 88.32         | GRU       | 7      | 32                       | 100            |
| 88.29         | GRU       | 8      | 5                        | 200            | no dropout 
| 88.03         | GRU       | >6     | 20                       | 40             | 0.3 (after Embedding and LSTM)
| 87.93         | GRU       | 4      | 32                       | 50             | 0.2 (after LSTM)
| 87.60         | GRU       | 5      | 5                        | 50             |  no dropout
| 87.5          | GRU       | 8      | 10                       | 20             |  no dropout
| 87.5          | GRU       | 5      | 32                       | 50             |
| 87.46         | GRU       | 8      | 16                       | 100            |
| < 87          | LSTM      | 9-11   | 32                       | 100            |
| 87.66         | GRU       | 5      | 32                       | 50             | 0.3 (after Embedding and LSTM)
| 86.5          | GRU       | >10    | 5                        | 10             | no dropout
Graphs
End of explanation
import re
words_only = r'[^\s!,.?\-":;0-9]+'
re.findall(words_only, "Some text to, tokenize. something's.Something-else?".lower())
def encode(reviewText):
    words = re.findall(words_only, reviewText.lower())
    reviewIDs = [start_char]
    for word in words:
        index = word_index.get(word, oov_char -index_from) + index_from # defaults to oov_char for missing
        if index > top_words:
            index = oov_char
        reviewIDs.append(index)  
    return reviewIDs
toText(encode('To code and back again. ikkyikyptangzooboing ni !!'))
# reviews from: 
# https://www.pluggedin.com/movie-reviews/solo-a-star-wars-story
# http://badmovie-badreview.com/category/bad-reviews/
user_reviews = ["This movie is horrible",
         "This wasn't a horrible movie and I liked it actually",
         "This movie was great.",
         "What a waste of time. It was too long and didn't make any sense.",
         "This was boring and drab.",
         "I liked the movie.",
         "I didn't like the movie.",
         "I like the lead actor but the movie as a whole fell flat",
         "I don't know. It was ok, some good and some bad. Some will like it, some will not like it.",
         "There are definitely heroic seeds at our favorite space scoundrel's core, though, seeds that simply need a little life experience to nurture them to growth. And that's exactly what this swooping heist tale is all about. You get a yarn filled with romance, high-stakes gambits, flashy sidekicks, a spunky robot and a whole lot of who's-going-to-outfox-who intrigue. Ultimately, it's the kind of colorful adventure that one could imagine Harrison Ford's version of Han recalling with a great deal of flourish … and a twinkle in his eye.",
         "There are times to be politically correct and there are times to write things about midget movies, and I’m afraid that sharing Ankle Biters with the wider world is an impossible task without taking the low road, so to speak. There are horrible reasons for this, all of them the direct result of the midgets that this film contains, which makes it sound like I am blaming midgets for my inability to regulate my own moral temperament but I like to think I am a…big…enough person (geddit?) to admit that the problem rests with me, and not the disabled.",
         "While Beowulf didn’t really remind me much of Beowulf, it did reminded me of something else. At first I thought it was Van Helsing, but that just wasn’t it. It only hit me when Beowulf finally told his backstory and suddenly even the dumbest of the dumb will realise that this is a simple ripoff of Blade. The badass hero, who is actually born from evil, now wants to destroy it, while he apparently has to fight his urges to become evil himself (not that it is mentioned beyond a single reference at the end of Beowulf) and even the music fits into the same range. Sadly Beowulf is not even nearly as interesting or entertaining as its role model. The only good aspects I can see in Beowulf would be the stupid beginning and Christopher Lamberts hair. But after those first 10 minutes, the movie becomes just boring and you don’t care much anymore.",
         "You don't frighten us, English pig-dogs!  Go and boil your bottoms, son of a silly person!  I blow my nose at you, so-called Arthur King!  You and all your silly English Knnnnnnnn-ighuts!!!"
        ]
X_user = np.array([encode(review) for review in user_reviews ])
X_user
X_user_pad = sequence.pad_sequences(X_user, maxlen=max_review_length)
X_user_pad
Explanation: Evaluate on Custom Text
End of explanation
for row in X_user_pad:
    print()
    print(toText(row))
Explanation: Features View
End of explanation
user_scores = model.predict(X_user_pad)
is_positive = user_scores >= 0.5  # I'm an optimist
for i in range(len(user_reviews)):
    print(  '\n%.2f %s:' % (user_scores[i][0], 'positive' if is_positive[i] else 'negative' ) + ' ' + user_reviews[i] )
Explanation: Results
End of explanation
<END_TASK>