<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 LSTM Recurrent Neural Network
Learning Objectives
Create map for converting IMDB dataset to readable reviews.
Create and build LSTM Recurrent Neural Network.
Visualise the Model and train the LSTM.
Evaluate model with test data and view results.
What is this?
This Jupyter Notebook contains Python code for building a LSTM Recurrent Neural Network that gives 87-88% accuracy on the IMDB Movie Review Sentiment Analysis Dataset. 
More information is given on this blogpost.
Introduction
Long Short Term Memory networks – usually just called “LSTMs” – are a special kind of RNN, capable of learning long-term dependencies. They were introduced by Hochreiter & Schmidhuber (1997), and were refined and popularized by many people in following work. They work tremendously well on a large variety of problems, and are now widely used.
LSTMs are explicitly designed to avoid the long-term dependency problem. Remembering information for long periods of time is practically their default behavior, not something they struggle to learn!
Each learning objective will correspond to a #TODO in the student lab notebook -- try to complete that notebook first before reviewing this solution notebook.
Setting up
When running this for the first time you may get a warning telling you to restart the Runtime. You can ignore this, but feel free to select "Kernel->Restart Kernel" from the overhead menu if you encounter problems.
Step1:  Note
Step2:  Create map for converting IMDB dataset to readable reviews
Reviews in the IMDB dataset have been encoded as a sequence of integers. Luckily the dataset also 
contains an index for converting the reviews back into human readable form.
Step3:  Data Insight
Here we take a closer look at our data. How many words do our reviews contain?
And what do our reviews look like in machine and human readable form?
Step4:  Pre-processing Data
We need to make sure that our reviews are of a uniform length. This is for the LSTM's parameters.
Some reviews will need to be truncated, while others need to be padded.
Step5:  Create and build LSTM Recurrent Neural Network
Step6:  Visualise the Model
Step7:  Train the LSTM
Step8:  Evaluate model with test data and view results
Step9:  View some incorrect predictions
Let's have a look at some of the incorrectly classified reviews. For readability we remove the padding.
Step10:  Run your own text against the trained model
This is a fun way to test out the limits of the trained model. To avoid getting errors - type in lower case only and do not use punctuation! 
You'll see the raw prediction from the model - basically a value between 0 and 1.
<ASSISTANT_TASK:>
Python Code:
# keras.datasets.imdb is broken in TensorFlow 1.13 and 1.14 due to numpy 1.16.3
!pip install numpy==1.16.2
# All the imports!
import tensorflow as tf 
import numpy as np
from tensorflow.keras.preprocessing import sequence
from numpy import array
# Supress deprecation warnings
import logging
logging.getLogger('tensorflow').disabled = True
# Fetch "IMDB Movie Review" data, constraining our reviews to 
# the 10000 most commonly used words
vocab_size = 10000
(x_train, y_train), (x_test, y_test) = tf.keras.datasets.imdb.load_data(num_words=vocab_size)
# Map for readable classnames
class_names = ["Negative", "Positive"]
Explanation: LSTM Recurrent Neural Network
Learning Objectives
Create map for converting IMDB dataset to readable reviews.
Create and build LSTM Recurrent Neural Network.
Visualise the Model and train the LSTM.
Evaluate model with test data and view results.
What is this?
This Jupyter Notebook contains Python code for building a LSTM Recurrent Neural Network that gives 87-88% accuracy on the IMDB Movie Review Sentiment Analysis Dataset. 
More information is given on this blogpost.
Introduction
Long Short Term Memory networks – usually just called “LSTMs” – are a special kind of RNN, capable of learning long-term dependencies. They were introduced by Hochreiter & Schmidhuber (1997), and were refined and popularized by many people in following work. They work tremendously well on a large variety of problems, and are now widely used.
LSTMs are explicitly designed to avoid the long-term dependency problem. Remembering information for long periods of time is practically their default behavior, not something they struggle to learn!
Each learning objective will correspond to a #TODO in the student lab notebook -- try to complete that notebook first before reviewing this solution notebook.
Setting up
When running this for the first time you may get a warning telling you to restart the Runtime. You can ignore this, but feel free to select "Kernel->Restart Kernel" from the overhead menu if you encounter problems.
End of explanation
# Show the currently installed version of TensorFlow
print("TensorFlow version: ",tf.version.VERSION)
Explanation: Note: Please ignore any incompatibility errors or warnings as it does not impact the notebook's functionality.
This notebook uses TF2.x.
Please check your tensorflow version using the cell below.
End of explanation
# Get the word index from the dataset
word_index = tf.keras.datasets.imdb.get_word_index()
# Ensure that "special" words are mapped into human readable terms 
word_index = {k:(v+3) for k,v in word_index.items()}
word_index["<PAD>"] = 0
word_index["<START>"] = 1
word_index["<UNKNOWN>"] = 2
word_index["<UNUSED>"] = 3
# Perform reverse word lookup and make it callable
# TODO
reverse_word_index = dict([(value, key) for (key, value) in word_index.items()])
def decode_review(text):
    return ' '.join([reverse_word_index.get(i, '?') for i in text])
Explanation: Create map for converting IMDB dataset to readable reviews
Reviews in the IMDB dataset have been encoded as a sequence of integers. Luckily the dataset also 
contains an index for converting the reviews back into human readable form.
End of explanation
# Concatenate test and training datasets
allreviews = np.concatenate((x_train, x_test), axis=0)
# Review lengths across test and training whole datasets
print("Maximum review length: {}".format(len(max((allreviews), key=len))))
print("Minimum review length: {}".format(len(min((allreviews), key=len))))
result = [len(x) for x in allreviews]
print("Mean review length: {}".format(np.mean(result)))
# Print a review and it's class as stored in the dataset. Replace the number
# to select a different review.
print("")
print("Machine readable Review")
print("  Review Text: " + str(x_train[60]))
print("  Review Sentiment: " + str(y_train[60]))
# Print a review and it's class in human readable format. Replace the number
# to select a different review.
print("")
print("Human Readable Review")
print("  Review Text: " + decode_review(x_train[60]))
print("  Review Sentiment: " + class_names[y_train[60]])
Explanation: Data Insight
Here we take a closer look at our data. How many words do our reviews contain?
And what do our reviews look like in machine and human readable form?
End of explanation
# The length of reviews
review_length = 500
# Padding / truncated our reviews
x_train = sequence.pad_sequences(x_train, maxlen = review_length)
x_test = sequence.pad_sequences(x_test, maxlen = review_length)
# Check the size of our datasets. Review data for both test and training should 
# contain 25000 reviews of 500 integers. Class data should contain 25000 values, 
# one for each review. Class values are 0 or 1, indicating a negative 
# or positive review.
print("Shape Training Review Data: " + str(x_train.shape))
print("Shape Training Class Data: " + str(y_train.shape))
print("Shape Test Review Data: " + str(x_test.shape))
print("Shape Test Class Data: " + str(y_test.shape))
# Note padding is added to start of review, not the end
print("")
print("Human Readable Review Text (post padding): " + decode_review(x_train[60]))
Explanation: Pre-processing Data
We need to make sure that our reviews are of a uniform length. This is for the LSTM's parameters.
Some reviews will need to be truncated, while others need to be padded.
End of explanation
# We begin by defining an empty stack. We'll use this for building our 
# network, later by layer.
model = tf.keras.models.Sequential()
# The Embedding Layer provides a spatial mapping (or Word Embedding) of all the 
# individual words in our training set. Words close to one another share context 
# and or meaning. This spatial mapping is learning during the training process.
model.add(
    tf.keras.layers.Embedding(
        input_dim = vocab_size, # The size of our vocabulary 
        output_dim = 32, # Dimensions to which each words shall be mapped
        input_length = review_length # Length of input sequences
    )
)
# Dropout layers fight overfitting and forces the model to learn multiple 
# representations of the same data by randomly disabling neurons in the 
# learning phase.
# TODO 
model.add(
    tf.keras.layers.Dropout(
        rate=0.25 # Randomly disable 25% of neurons
    )
)
# We are using a fast version of LSTM which is optimised for GPUs. This layer 
# looks at the sequence of words in the review, along with their word embeddings
# and uses both of these to determine the sentiment of a given review.
# TODO 
model.add(
    tf.keras.layers.LSTM(
        units=32 # 32 LSTM units in this layer
    )
)
# Add a second dropout layer with the same aim as the first.
# TODO 
model.add(
    tf.keras.layers.Dropout(
        rate=0.25 # Randomly disable 25% of neurons
    )
)
# All LSTM units are connected to a single node in the dense layer. A sigmoid 
# activation function determines the output from this node - a value 
# between 0 and 1. Closer to 0 indicates a negative review. Closer to 1 
# indicates a positive review.
model.add(
    tf.keras.layers.Dense(
        units=1, # Single unit
        activation='sigmoid' # Sigmoid activation function (output from 0 to 1)
    )
)
# Compile the model
model.compile(
    loss=tf.keras.losses.binary_crossentropy, # loss function
    optimizer=tf.keras.optimizers.Adam(), # optimiser function
    metrics=['accuracy']) # reporting metric
# Display a summary of the models structure
model.summary()
Explanation: Create and build LSTM Recurrent Neural Network
End of explanation
tf.keras.utils.plot_model(model, to_file='model.png', show_shapes=True, show_layer_names=False)
Explanation: Visualise the Model
End of explanation
# Train the LSTM on the training data
history = model.fit(
    # Training data : features (review) and classes (positive or negative)
    x_train, y_train,
                    
    # Number of samples to work through before updating the 
    # internal model parameters via back propagation. The 
    # higher the batch, the more memory you need.
    batch_size=256, 
    # An epoch is an iteration over the entire training data.
    epochs=3, 
    
    # The model will set apart his fraction of the training 
    # data, will not train on it, and will evaluate the loss
    # and any model metrics on this data at the end of 
    # each epoch.
    validation_split=0.2,
    
    verbose=1
) 
Explanation: Train the LSTM
End of explanation
# Get Model Predictions for test data
# TODO 
from sklearn.metrics import classification_report
predicted_classes = np.argmax(model.predict(x_test), axis=-1)
print(classification_report(y_test, predicted_classes, target_names=class_names))
Explanation: Evaluate model with test data and view results
End of explanation
predicted_classes_reshaped = np.reshape(predicted_classes, 25000)
incorrect = np.nonzero(predicted_classes_reshaped!=y_test)[0]
# We select the first 10 incorrectly classified reviews
for j, incorrect in enumerate(incorrect[0:20]):
    
    predicted = class_names[predicted_classes_reshaped[incorrect]]
    actual = class_names[y_test[incorrect]]
    human_readable_review = decode_review(x_test[incorrect])
    
    print("Incorrectly classified Test Review ["+ str(j+1) +"]") 
    print("Test Review #" + str(incorrect)  + ": Predicted ["+ predicted + "] Actual ["+ actual + "]")
    print("Test Review Text: " + human_readable_review.replace("<PAD> ", ""))
    print("")
Explanation: View some incorrect predictions
Let's have a look at some of the incorrectly classified reviews. For readability we remove the padding.
End of explanation
# Write your own review
review = "this was a terrible film with too much sex and violence i walked out halfway through"
#review = "this is the best film i have ever seen it is great and fantastic and i loved it"
#review = "this was an awful film that i will never see again"
# Encode review (replace word with integers)
tmp = []
for word in review.split(" "):
    tmp.append(word_index[word])
# Ensure review is 500 words long (by padding or truncating)
tmp_padded = sequence.pad_sequences([tmp], maxlen=review_length) 
# Run your processed review against the trained model
rawprediction = model.predict(array([tmp_padded][0]))[0][0]
prediction = int(round(rawprediction))
# Test the model and print the result
print("Review: " + review)
print("Raw Prediction: " + str(rawprediction))
print("Predicted Class: " + class_names[prediction])
Explanation: Run your own text against the trained model
This is a fun way to test out the limits of the trained model. To avoid getting errors - type in lower case only and do not use punctuation! 
You'll see the raw prediction from the model - basically a value between 0 and 1.
End of explanation
<END_TASK>