<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 <h1 style="text-align
Step1:  Next-word prediction task
Part 1
Step2:  1.2.Symbols encoding
The LSTM input's can only be numbers. A way to convert words (symbols or any items) to numbers is to assign a unique integer to each word. This process is often based on frequency of occurrence for efficient coding purpose.
Here, we define a function to build an indexed word dictionary (word->number). The "build_vocabulary" function builds both
Step3:  Run the cell below to display the vocabulary
Step4:  Part 2 
Step5:  Training Parameters and constants
Step6:  Define the Loss/Cost and optimizer
Step7:  Comment
Step8:  Part 3 
Step9:  Comment
Step10:  Comment
Step11:  Comment
Step12:  3.3. Play with number of inputs
The number of input in our example is 3, see what happens when you use other number (1 and 5)
n_input = 1
Step13:  Comment
Step14:  Comment
<ASSISTANT_TASK:>
Python Code:
import numpy as np
import collections # used to build the dictionary
import random
import time
from time import time
import pickle # may be used to save your model 
import matplotlib.pyplot as plt
#Import Tensorflow and rnn
import tensorflow as tf
from tensorflow.contrib import rnn  
# Target log path
logs_path = 'lstm_words'
writer = tf.summary.FileWriter(logs_path)
Explanation: <h1 style="text-align:center">Deep Learning   </h1>
<h1 style="text-align:center"> Lab Session 3 - 3 Hours </h1>
<h1 style="text-align:center">Long Short Term Memory (LSTM) for Language Modeling</h1>
<b> Student 1:</b> CANALE
<b> Student 2:</b> ELLENA
In this Lab Session,  you will build and train a Recurrent Neural Network, based on Long Short-Term Memory (LSTM) units for next word prediction task. 
Answers and experiments should be made by groups of one or two students. Each group should fill and run appropriate notebook cells. 
Once you have completed all of the code implementations and successfully answered each question above, you may finalize your work by exporting the iPython Notebook as an pdf document using print as PDF (Ctrl+P). Do not forget to run all your cells before generating your final report and do not forget to include the names of all participants in the group. The lab session should be completed by June 9th 2017.
Send you pdf file to benoit.huet@eurecom.fr and olfa.ben-ahmed@eurecom.fr using [DeepLearning_lab3] as Subject of your email.
Introduction
You will train a LSTM to predict the next word using a sample short story. The LSTM will learn to predict the next item of a sentence from the 3 previous items (given as input). Ponctuation marks are considered as dictionary items so they can be predicted too. Figure 1 shows the LSTM and the process of next word prediction. 
<img src="lstm.png" height="370" width="370"> 
Each word (and ponctuation) from text sentences is encoded by a unique integer. The integer value corresponds to the index of the corresponding word (or punctuation mark) in the dictionnary. The network output is a one-hot-vector indicating the index of the predicted word in the reversed dictionary (Section 1.2). For example if the prediction is 86, the predicted word will be "company". 
You will use a sample short story from Aesop’s Fables (http://www.taleswithmorals.com/) to train your model. 
<font size="3" face="verdana" > <i> "There was once a young Shepherd Boy who tended his sheep at the foot of a mountain near a dark forest.
It was rather lonely for him all day, so he thought upon a plan by which he could get a little company and some excitement.
He rushed down towards the village calling out "Wolf, Wolf," and the villagers came out to meet him, and some of them stopped with him for a considerable time.
This pleased the boy so much that a few days afterwards he tried the same trick, and again the villagers came to his help.
But shortly after this a Wolf actually did come out from the forest, and began to worry the sheep, and the boy of course cried out "Wolf, Wolf," still louder than before.
But this time the villagers, who had been fooled twice before, thought the boy was again deceiving them, and nobody stirred to come to his help.
So the Wolf made a good meal off the boy's flock, and when the boy complained, the wise man of the village said:
"A liar will not be believed, even when he speaks the truth."  "</i> </font>.    
Start by loading the necessary libraries and resetting the default computational graph. For more details about the rnn packages, we suggest you to take a look at https://www.tensorflow.org/api_guides/python/contrib.rnn
End of explanation
def load_data(filename):
    with open(filename) as f:
        data = f.readlines()
    data = [x.strip().lower() for x in data]
    data = [data[i].split() for i in range(len(data))]
    data = np.array(data)
    data = np.reshape(data, [-1, ])
    print(data)
    return data
#Run the cell 
train_file ='data/story.txt'
train_data = load_data(train_file)
print("Loaded training data...")
print(len(train_data))
Explanation: Next-word prediction task
Part 1: Data  preparation
1.1. Loading data
Load and split the text of our story
End of explanation
def build_vocabulary(words):
    count = collections.Counter(words).most_common()
    dic= dict()
    for word, _ in count:
        dic[word] = len(dic)
    reverse_dic= dict(zip(dic.values(), dic.keys()))
    return dic, reverse_dic
Explanation: 1.2.Symbols encoding
The LSTM input's can only be numbers. A way to convert words (symbols or any items) to numbers is to assign a unique integer to each word. This process is often based on frequency of occurrence for efficient coding purpose.
Here, we define a function to build an indexed word dictionary (word->number). The "build_vocabulary" function builds both:
Dictionary : used for encoding words to numbers for the LSTM inputs 
Reverted dictionnary : used for decoding the outputs of the LSTM into words (and punctuation).
For example, in the story above, we have 113 individual words. The "build_vocabulary" function builds a dictionary with the following entries ['the': 0], [',': 1], ['company': 85],...
End of explanation
dictionary, reverse_dictionary = build_vocabulary(train_data)
vocabulary_size= len(dictionary) 
print "Dictionary size (Vocabulary size) = ", vocabulary_size
print("\n")
print("Dictionary : \n")
print(dictionary)
print("\n")
print("Reverted Dictionary : \n" )
print(reverse_dictionary)
Explanation: Run the cell below to display the vocabulary
End of explanation
def lstm_model(x, w, b, n_input, n_hidden):
    # reshape to [1, n_input]
    x = tf.reshape(x, [-1, n_input])
    # Generate a n_input-element sequence of inputs
    # (eg. [had] [a] [general] -> [20] [6] [33])
    x = tf.split(x,n_input,1)
    # 1-layer LSTM with n_hidden units.
    rnn_cell = rnn.BasicLSTMCell(n_hidden)
    
    #improvement
    #rnn_cell = rnn.MultiRNNCell([rnn.BasicLSTMCell(n_hidden),rnn.BasicLSTMCell(n_hidden)])
    #rnn_cell = rnn.MultiRNNCell([rnn.BasicLSTMCell(n_hidden),rnn.BasicLSTMCell(n_hidden),rnn.BasicLSTMCell(n_hidden)])
    # generate prediction
    outputs, states = rnn.static_rnn(rnn_cell, x, dtype=tf.float32)
    # there are n_input outputs but
    # we only want the last output
    return tf.matmul(outputs[-1], w['out']) + b['out']
Explanation: Part 2 : LSTM Model in TensorFlow
Since you have defined how the data will be modeled, you are now to develop an LSTM model to predict the word of following a sequence of 3 words. 
2.1. Model definition
Define a 2-layers LSTM model.  
For this use the following classes from the tensorflow.contrib library:
rnn.BasicLSTMCell(number of hidden units) 
rnn.static_rnn(rnn_cell, data, dtype=tf.float32)
rnn.MultiRNNCell(,)
You may need some tensorflow functions (https://www.tensorflow.org/api_docs/python/tf/) :
- tf.split
- tf.reshape 
- ...
End of explanation
# Training Parameters
learning_rate = 0.001
epochs = 50000
display_step = 1000
n_input = 3
#For each LSTM cell that you initialise, supply a value for the hidden dimension, number of units in LSTM cell
n_hidden = 64
# tf Graph input
x = tf.placeholder("float", [None, n_input, 1])
y = tf.placeholder("float", [None, vocabulary_size])
# LSTM  weights and biases
weights = { 'out': tf.Variable(tf.random_normal([n_hidden, vocabulary_size]))}
biases = {'out': tf.Variable(tf.random_normal([vocabulary_size])) }
#build the model
pred = lstm_model(x, weights, biases,n_input,n_hidden)
Explanation: Training Parameters and constants
End of explanation
# Loss and optimizer
cost = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(logits=pred, labels=y))
#cost = tf.reduce_mean(-tf.reduce_sum(y*tf.log(pred), reduction_indices=1))
#cost = tf.reduce_mean(-tf.reduce_sum(y*tf.log(tf.clip_by_value(pred,-1.0,1.0)), reduction_indices=1))
optimizer = tf.train.RMSPropOptimizer(learning_rate).minimize(cost)
# Model evaluation
correct_pred = tf.equal(tf.argmax(pred,1), tf.argmax(y,1))
accuracy = tf.reduce_mean(tf.cast(correct_pred, tf.float32))
Explanation: Define the Loss/Cost and optimizer
End of explanation
#run the cell
def test(sentence, session, verbose=False):
    sentence = sentence.strip()
    words = sentence.split(' ')
    if len(words) != n_input:
        print("sentence length should be equel to", n_input, "!")
    try:
        symbols_inputs = [dictionary[str(words[i - n_input])] for i in range(n_input)]
        keys = np.reshape(np.array(symbols_inputs), [-1, n_input, 1])
        onehot_pred = session.run(pred, feed_dict={x: keys})
        onehot_pred_index = int(tf.argmax(onehot_pred, 1).eval())
        words.append(reverse_dictionary[onehot_pred_index])
        sentence = " ".join(words)
        if verbose:
            print(sentence)
        return reverse_dictionary[onehot_pred_index]
    except:
        print " ".join(["Word", words[i - n_input], "not in dictionary"])
Explanation: Comment:
We decided to apply the softmax and calculate the cost at the same time. In this way we can use the method softmax_cross_entropy_with_logits, which is more numerically stable in corner cases than applying the softmax and then calculating the cross entropy
We give you here the Test Function
End of explanation
# Initializing the variables
init = tf.global_variables_initializer()
saver = tf.train.Saver()
start_time = time()
# Launch the graph
with tf.Session() as session:
    session.run(init)
    step = 0
    offset = random.randint(0,n_input+1)
    end_offset = n_input + 1
    acc_total = 0
    loss_total = 0
    writer.add_graph(session.graph)
    while step < epochs:
        # Generate a minibatch. Add some randomness on selection process.
        if offset > (len(train_data)-end_offset):
            offset = random.randint(0, n_input+1)
        symbols_in_keys = [ [dictionary[ str(train_data[i])]] for i in range(offset, offset+n_input) ]
        symbols_in_keys = np.reshape(np.array(symbols_in_keys), [-1, n_input, 1])
        symbols_out_onehot = np.zeros([len(dictionary)], dtype=float)
        symbols_out_onehot[dictionary[str(train_data[offset+n_input])]] = 1.0
        symbols_out_onehot = np.reshape(symbols_out_onehot,[1,-1])
        _, acc, loss, onehot_pred = session.run([optimizer, accuracy, cost, pred], \
                                                feed_dict={x: symbols_in_keys, y: symbols_out_onehot})
        loss_total += loss
        acc_total += acc
        if (step+1) % display_step == 0:
            print("Iter= " + str(step+1) + ", Average Loss= " + \
                  "{:.6f}".format(loss_total/display_step) + ", Average Accuracy= " + \
                  "{:.2f}%".format(100*acc_total/display_step))
            acc_total = 0
            loss_total = 0
            symbols_in = [train_data[i] for i in range(offset, offset + n_input)]
            symbols_out = train_data[offset + n_input]
            symbols_out_pred = reverse_dictionary[int(tf.argmax(onehot_pred, 1).eval())]
            print("%s - [%s] vs [%s]" % (symbols_in,symbols_out,symbols_out_pred))
        step += 1
        offset += (n_input+1)
    print("Optimization Finished!")
    print("Elapsed time: ", time() - start_time)
    print("Run on command line.")
    print("\ttensorboard --logdir=%s" % (logs_path))
    print("Point your web browser to: http://localhost:6006/")
    save_path = saver.save(session, "model.ckpt")
    print("Model saved in file: %s" % save_path)
Explanation: Part 3 : LSTM Training
In the Training process, at each epoch, 3 words are taken from the training data, encoded to integer to form the input vector. The training labels are one-hot vector encoding the word that comes after the 3 inputs words. Display the loss and the training accuracy every 1000 iteration. Save the model at the end of training in the lstm_model folder
End of explanation
with tf.Session() as sess:
    # Initialize variables
    sess.run(init)
    # Restore model weights from previously saved model
    saver.restore(sess, "./model.ckpt")
    print(test('get a little', sess))
    print(test('nobody tried to', sess))
Explanation: Comment:
We created different models with different number of layers, and we have seen that the best accuracy is achieved using only 2 laers. Using more or less layers we achieve a lower accuracy
Part 4 : Test your model
3.1. Next word prediction
Load your model (using the model_saved variable given in the training session) and test the sentences :
- 'get a little' 
- 'nobody tried to'
- Try with other sentences using words from the stroy's vocabulary.
End of explanation
#Your implementation goes here 
with tf.Session() as sess:
    # Initialize variables
    sess.run(init)
    # Restore model weights from previously saved model
    saver.restore(sess, "./model.ckpt")
    
    #a sentence is concluded when we find a dot.
    fable = [random.choice(dictionary.keys()) for _ in range(3)]
    n_sentences = fable.count('.')
    offset = 0
    while n_sentences < 5:
        next_word = test(' '.join(fable[offset:offset+3]), sess)
        fable.append(next_word)
        if next_word == '.':
            n_sentences += 1
        offset+=1
    print(' '.join(fable))
Explanation: Comment:
Here it looks that the RNN is working, in fact it can predict correctly the next word. 
We should not that in this case is difficult to check if the RNN is actually overfitting the training data.
3.2. More fun with the Fable Writer !
You will use the RNN/LSTM model learned in the previous question to create a
new story/fable.
For this you will choose 3 words from the dictionary which will start your
story and initialize your network. Using those 3 words the RNN will generate
the next word or the story. Using the last 3 words (the newly predicted one
and the last 2 from the input) you will use the network to predict the 5
word of the story.. and so on until your story is 5 sentence long. 
Make a point at the end of your story. 
To implement that, you will use the test function. 
This is the original fable, we will look at it to note an eventual overfitting
It was rather lonely for him all day, so he thought upon a plan by which he could get a little company and some excitement.
He rushed down towards the village calling out "Wolf, Wolf," and the villagers came out to meet him, and some of them stopped with him for a considerable time.
This pleased the boy so much that a few days afterwards he tried the same trick, and again the villagers came to his help.
But shortly after this a Wolf actually did come out from the forest, and began to worry the sheep, and the boy of course cried out "Wolf, Wolf," still louder than before.
But this time the villagers, who had been fooled twice before, thought the boy was again deceiving them, and nobody stirred to come to his help.
So the Wolf made a good meal off the boy's flock, and when the boy complained, the wise man of the village said:
"A liar will not be believed, even when he speaks the truth.
End of explanation
def load_data(filename):
    with open(filename) as f:
        data = f.readlines()
    data = [x.strip().lower() for x in data]
    data = [data[i].split() for i in range(len(data))]
    data = np.array(data)
    data = np.reshape(data, [-1, ])
    return data
train_file ='data/story.txt'
train_data = load_data(train_file)
def build_vocabulary(words):
    count = collections.Counter(words).most_common()
    dic= dict()
    for word, _ in count:
        dic[word] = len(dic)
    reverse_dic= dict(zip(dic.values(), dic.keys()))
    return dic, reverse_dic
dictionary, reverse_dictionary = build_vocabulary(train_data)
vocabulary_size= len(dictionary) 
import numpy as np
import collections # used to build the dictionary
import random
import time
from time import time
import pickle # may be used to save your model 
import matplotlib.pyplot as plt
#Import Tensorflow and rnn
import tensorflow as tf
from tensorflow.contrib import rnn  
def create_train_model(n_input = 3, n_layers = 2,verbose = False):
    tf.reset_default_graph()
    # Target log path
    logs_path = 'lstm_words'
    writer = tf.summary.FileWriter(logs_path)
    
    def lstm_model(x, w, b, n_input, n_hidden,n_layers):
        # reshape to [1, n_input]
        x = tf.reshape(x, [-1, n_input])
        # Generate a n_input-element sequence of inputs
        # (eg. [had] [a] [general] -> [20] [6] [33])
        x = tf.split(x,n_input,1)
        rnn_layers = [rnn.BasicLSTMCell(n_hidden) for _ in range(n_layers)]
        rnn_cell = rnn.MultiRNNCell(rnn_layers)
        # generate prediction
        outputs, states = rnn.static_rnn(rnn_cell, x, dtype=tf.float32)
        # there are n_input outputs but
        # we only want the last output
        return tf.matmul(outputs[-1], w['out']) + b['out']
    
    # Training Parameters
    learning_rate = 0.001
    epochs = 50000
    display_step = 1000
    #For each LSTM cell that you initialise, supply a value for the hidden dimension, number of units in LSTM cell
    n_hidden = 64
    # tf Graph input
    x = tf.placeholder("float", [None, n_input, 1])
    y = tf.placeholder("float", [None, vocabulary_size])
    # LSTM  weights and biases
    weights = { 'out': tf.Variable(tf.random_normal([n_hidden, vocabulary_size]))}
    biases = {'out': tf.Variable(tf.random_normal([vocabulary_size])) }
    #build the model
    pred = lstm_model(x, weights, biases,n_input,n_hidden,n_layers)
    # Loss and optimizer
    cost = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(logits=pred, labels=y))
    #cost = tf.reduce_mean(-tf.reduce_sum(y*tf.log(pred), reduction_indices=1))
    #cost = tf.reduce_mean(-tf.reduce_sum(y*tf.log(tf.clip_by_value(pred,-1.0,1.0)), reduction_indices=1))
    optimizer = tf.train.RMSPropOptimizer(learning_rate).minimize(cost)
    # Model evaluation
    correct_pred = tf.equal(tf.argmax(pred,1), tf.argmax(y,1))
    accuracy = tf.reduce_mean(tf.cast(correct_pred, tf.float32))
    
    # Initializing the variables
    init = tf.global_variables_initializer()
    saver = tf.train.Saver()
    start_time = time()
    # Launch the graph
    with tf.Session() as session:
        session.run(init)
        step = 0
        offset = random.randint(0,n_input+1)
        end_offset = n_input + 1
        acc_total = 0
        loss_total = 0
        writer.add_graph(session.graph)
        while step < epochs:
            # Generate a minibatch. Add some randomness on selection process.
            if offset > (len(train_data)-end_offset):
                offset = random.randint(0, n_input+1)
            symbols_in_keys = [ [dictionary[ str(train_data[i])]] for i in range(offset, offset+n_input) ]
            symbols_in_keys = np.reshape(np.array(symbols_in_keys), [-1, n_input, 1])
            symbols_out_onehot = np.zeros([len(dictionary)], dtype=float)
            symbols_out_onehot[dictionary[str(train_data[offset+n_input])]] = 1.0
            symbols_out_onehot = np.reshape(symbols_out_onehot,[1,-1])
            _, acc, loss, onehot_pred = session.run([optimizer, accuracy, cost, pred], \
                                                    feed_dict={x: symbols_in_keys, y: symbols_out_onehot})
            loss_total += loss
            acc_total += acc
            if (step+1) % display_step == 0:
                if verbose or step+1 == epochs: print("Iter= " + str(step+1) + ", Average Loss= " + \
                      "{:.6f}".format(loss_total/display_step) + ", Average Accuracy= " + \
                      "{:.2f}%".format(100*acc_total/display_step))
                acc_total = 0
                loss_total = 0
                symbols_in = [train_data[i] for i in range(offset, offset + n_input)]
                symbols_out = train_data[offset + n_input]
                symbols_out_pred = reverse_dictionary[int(tf.argmax(onehot_pred, 1).eval())]
                if verbose: print("%s - [%s] vs [%s]" % (symbols_in,symbols_out,symbols_out_pred))
            step += 1
            offset += (n_input+1)
            
        
        print("Optimization Finished!")
        print("Elapsed time: ", time() - start_time)
        print("Run on command line.")
        print("\ttensorboard --logdir=%s" % (logs_path))
        print("Point your web browser to: http://localhost:6006/")
        save_path = saver.save(session, "model.ckpt")
        print("Model saved in file: %s" % save_path)
        
        #run the cell
        def test(sentence, session, verbose=False):
            sentence = sentence.strip()
            words = sentence.split(' ')
            if len(words) != n_input:
                print("sentence length should be equel to", n_input, "!")
            try:
                symbols_inputs = [dictionary[str(words[i - n_input])] for i in range(n_input)]
                keys = np.reshape(np.array(symbols_inputs), [-1, n_input, 1])
                onehot_pred = session.run(pred, feed_dict={x: keys})
                onehot_pred_index = int(tf.argmax(onehot_pred, 1).eval())
                words.append(reverse_dictionary[onehot_pred_index])
                sentence = " ".join(words)
                if verbose:
                    print(sentence)
                return reverse_dictionary[onehot_pred_index]
            except:
                print " ".join(["Word", words[i - n_input], "not in dictionary"])
        
        #a sentence is concluded when we find a dot.
        fable = [random.choice(dictionary.keys()) for _ in range(n_input)]
        #print(dictionary)
        #print(fable)
        n_sentences = fable.count('.')
        offset = 0
        while n_sentences < 5 and len(fable) < 200:
            next_word = test(' '.join(fable[offset:offset+n_input]), session)
            fable.append(next_word)
            if next_word == '.':
                n_sentences += 1
            offset+=1
        print(' '.join(fable))
Explanation: Comment:
This is interesting, we see that the sentences have some sort of sense, but when we reach a point, we see the same sentence repated many times. Thus is probably due to overfitting, we should look more deeply. We see that the repeated sentence is different from the original one, but it is still always the same. We think this is due to the fact that the dot start always the same sentence. Maybe we could create more layers and see what happens.
End of explanation
create_train_model(n_input = 1, n_layers = 1)
create_train_model(n_input = 1, n_layers = 2)
create_train_model(n_input = 1, n_layers = 3)
Explanation: 3.3. Play with number of inputs
The number of input in our example is 3, see what happens when you use other number (1 and 5)
n_input = 1
End of explanation
create_train_model(n_input = 3, n_layers = 1)
create_train_model(n_input = 3, n_layers = 2)
create_train_model(n_input = 3, n_layers = 3)
Explanation: Comment:
Here we see that when the input size is 1 we obtain a vad model regardless of the number of layers, this is because we are basically predicting a word based on the preceding word. This not enough to create a sentence with some sort of sense.Looking ath the prediction accuracy, it is very low.
n_input = 3
End of explanation
create_train_model(n_input = 5, n_layers = 1)
create_train_model(n_input = 5, n_layers = 2)
create_train_model(n_input = 5, n_layers = 3)
Explanation: Comment:
Here we see some sentences that have a sense, but we see a tendency to repeat the sentence of the training fable. This is interesting, because during the training the single triples where chosen randomly and not sequentially. Somehow, the net learned the training fable.
n_input = 5
End of explanation
<END_TASK>