<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 <center> Text generation using RNN/LSTM (Character-level)</center>
<div class="alert alert-block alert-info">
<font size = 3><strong>In this notebook you will learn the How to use TensorFlow for create a Recurrent Neural Network</strong></font>
<br>    
- <a href="#intro">Introduction</a>
<br>
- <p><a href="#arch">Architectures</a></p>
    - <a href="#lstm">Long Short-Term Memory Model (LSTM)</a>
- <p><a href="#build">Building a LSTM with TensorFlow</a></p>
</div>
This code implements a Recurrent Neural Network with LSTM/RNN units for training/sampling from character-level language models. In other words the model takes a text file as input and trains the RNN network that learns to predict the next character in a sequence.
The RNN can then be used to generate text character by character that will look like the original training data. 
This code is based on this blog, and the code is an step-by-step implimentation of the character-level implimentation.
Step1:  Data loader
The following cell is a class that help to read data from input file.
Step2:  Parameters
Batch, number_of_batch, batch_size and seq_length
what is batch, number_of_batch, batch_size and seq_length in the charcter level example?  
Lets assume the input is 'here is an example'. Then
Step3:  LSTM Architecture
each LSTM cell has an input layre, which its size is 128 units. 
128 is dimensionality of embedding vector.
rnn_size = num_units = num_hidden_units
Step4:  Input and output
Step5:  Defining stacked RNN Cell
BasicRNNCell is the most basic RNN cell.
Step6:  The memory state of the network is initialized with a vector of zeros and gets updated after reading each character.
BasicRNNCell.zero_state(batch_size, dtype) Return zero-filled state tensor(s).
Args
Step7:  Embedding
Step8:  Feeding a batch of 50 sequence to a RNN
Step9:  outputs is 50x[60*128]. We need to reshape it to [60x50x128]. Then we can calculate the softmax
Step10:  Using classes
Now that we have learned how the networks work, we can put all together
Step11:  the input is always a matrix of of shape [n x m]. Where n is the batch size, m is the feature size. 
In our case, the input shape will be [60 x ??]. 
size of data is 1113000, number of batches are 371, batch size is 60 and sequence length is 50. so, 5060371= 1113000
we have 50 epochs. 
each input matrix will represent 1 update per epoch.
Creating the LSTM object
Step12:  Train usinng LSTMModel class
Step13:  Sample
Step14:  Sample using function
<ASSISTANT_TASK:>
Python Code:
import tensorflow as tf
import time
print('TensorFlow version:', tf.__version__)
Explanation: <center> Text generation using RNN/LSTM (Character-level)</center>
<div class="alert alert-block alert-info">
<font size = 3><strong>In this notebook you will learn the How to use TensorFlow for create a Recurrent Neural Network</strong></font>
<br>    
- <a href="#intro">Introduction</a>
<br>
- <p><a href="#arch">Architectures</a></p>
    - <a href="#lstm">Long Short-Term Memory Model (LSTM)</a>
- <p><a href="#build">Building a LSTM with TensorFlow</a></p>
</div>
This code implements a Recurrent Neural Network with LSTM/RNN units for training/sampling from character-level language models. In other words the model takes a text file as input and trains the RNN network that learns to predict the next character in a sequence.
The RNN can then be used to generate text character by character that will look like the original training data. 
This code is based on this blog, and the code is an step-by-step implimentation of the character-level implimentation.
End of explanation
import codecs
import os
import collections
from six.moves import cPickle
import numpy as np
class TextLoader():
    def __init__(self, data_dir, batch_size, seq_length, encoding='utf-8'):
        self.data_dir = data_dir
        self.batch_size = batch_size
        self.seq_length = seq_length
        self.encoding = encoding
        input_file = os.path.join(data_dir, "input.txt")
        vocab_file = os.path.join(data_dir, "vocab.pkl")
        tensor_file = os.path.join(data_dir, "data.npy")
        if not (os.path.exists(vocab_file) and os.path.exists(tensor_file)):
            print("reading text file")
            self.preprocess(input_file, vocab_file, tensor_file)
        else:
            print("loading preprocessed files")
            self.load_preprocessed(vocab_file, tensor_file)
        self.create_batches()
        self.reset_batch_pointer()
    def preprocess(self, input_file, vocab_file, tensor_file):
        with codecs.open(input_file, "r", encoding=self.encoding) as f:
            data = f.read()
        counter = collections.Counter(data)
        count_pairs = sorted(counter.items(), key=lambda x: -x[1])
        self.chars, _ = zip(*count_pairs)
        self.vocab_size = len(self.chars)
        self.vocab = dict(zip(self.chars, range(len(self.chars))))
        with open(vocab_file, 'wb') as f:
            cPickle.dump(self.chars, f)
        self.tensor = np.array(list(map(self.vocab.get, data)))
        np.save(tensor_file, self.tensor)
    def load_preprocessed(self, vocab_file, tensor_file):
        with open(vocab_file, 'rb') as f:
            self.chars = cPickle.load(f)
        self.vocab_size = len(self.chars)
        self.vocab = dict(zip(self.chars, range(len(self.chars))))
        self.tensor = np.load(tensor_file)
        self.num_batches = int(self.tensor.size / (self.batch_size *
                                                   self.seq_length))
    def create_batches(self):
        self.num_batches = int(self.tensor.size / (self.batch_size *
                                                   self.seq_length))
        # When the data (tensor) is too small, let's give them a better error message
        if self.num_batches==0:
            assert False, "Not enough data. Make seq_length and batch_size small."
        self.tensor = self.tensor[:self.num_batches * self.batch_size * self.seq_length]
        xdata = self.tensor
        ydata = np.copy(self.tensor)
        ydata[:-1] = xdata[1:]
        ydata[-1] = xdata[0]
        self.x_batches = np.split(xdata.reshape(self.batch_size, -1), self.num_batches, 1)
        self.y_batches = np.split(ydata.reshape(self.batch_size, -1), self.num_batches, 1)
    def next_batch(self):
        x, y = self.x_batches[self.pointer], self.y_batches[self.pointer]
        self.pointer += 1
        return x, y
    def reset_batch_pointer(self):
        self.pointer = 0
Explanation: Data loader
The following cell is a class that help to read data from input file.
End of explanation
batch_size = 60 #minibatch size, i.e. size of dataset in each epoch
seq_length = 50 #RNN sequence length
num_epochs = 25 # you should change it to 50 if you want to see a relatively good results
learning_rate = 0.002
decay_rate = 0.97
rnn_size = 128 #size of RNN hidden state
num_layers = 2 #number of layers in the RNN
Explanation: Parameters
Batch, number_of_batch, batch_size and seq_length
what is batch, number_of_batch, batch_size and seq_length in the charcter level example?  
Lets assume the input is 'here is an example'. Then:
- txt_length = 18
- seq_length = 3
- batch_size = 2
- number_of_batch = 18/3*2 = 3
- batch = array (['h','e','r'],['e',' ','i'])
- sample Seq = 'her'  
So, what are our actual parameters?
End of explanation
!mkdir -p ../../data/character_model
!wget -nv -O ../../data/character_model/input.txt https://ibm.box.com/shared/static/a3f9e9mbpup09toq35ut7ke3l3lf03hg.txt 
data_loader = TextLoader('../../data/character_model/', batch_size, seq_length)
vocab_size = data_loader.vocab_size
data_loader.vocab_size
data_loader.num_batches
Explanation: LSTM Architecture
each LSTM cell has an input layre, which its size is 128 units. 
128 is dimensionality of embedding vector.
rnn_size = num_units = num_hidden_units:   = LSTM size
Each LSTM cell has a hidden layer, where there are some hidden units.
The argument n_hidden=128 of BasicLSTMCell is the number of hidden units of the LSTM (inside A).
Each LSTM cell keeps a vector, called hidden state vector, of size n_hidden=128.
A hidden state vector; which is the memory of the LSTM, accumulates using its (forget, input, and output) gates through time. 
For each LSTM cell that we initialise, we need to supply a value (128 in this case) for the hidden dimension, or as some people like to call it, the number of units in the LSTM cell. 
"num_units" is equivalant to "size of RNN hidden state"
rnn_size= 128, is also the dimension size of W2V/embedding, for each character/word.
An LSTM keeps two pieces of information as it propagates through time: 
A hidden state vector
A previous time-step output
To make the name num_units more intuitive, you can think of it as the number of hidden units in the LSTM cell, or the number of memory units in the cell.
number of hidden units is the dimensianality of the output (= dimesianality of the state) of the LSTM cell.
num_layers = 2
number of layers in the RNN
An input of MultiRNNCell is cells which is list of RNNCells that will be composed in this order.
End of explanation
x,y = data_loader.next_batch()
x
x.shape  #batch_size=60, seq_length=50
y
print('Vocabulary size:', data_loader.vocab_size)
print(", ".join(sorted(list(data_loader.chars))))
data_loader.vocab['t']
Explanation: Input and output
End of explanation
# a two layer cell
with tf.variable_scope('multi_rnn_cell'):
    stacked_cell = tf.contrib.rnn.MultiRNNCell([tf.contrib.rnn.BasicRNNCell(rnn_size) for _ in range(num_layers)])
# hidden state size
stacked_cell.output_size
stacked_cell.state_size
input_data = tf.placeholder(tf.int32, [batch_size, seq_length])# a 60x50
targets = tf.placeholder(tf.int32, [batch_size, seq_length]) # a 60x50
Explanation: Defining stacked RNN Cell
BasicRNNCell is the most basic RNN cell.
End of explanation
initial_state = stacked_cell.zero_state(batch_size, tf.float32) #why batch_size ? 60x128
input_data
session = tf.Session()
feed_dict={input_data:x, targets:y}
session.run(input_data, feed_dict)
Explanation: The memory state of the network is initialized with a vector of zeros and gets updated after reading each character.
BasicRNNCell.zero_state(batch_size, dtype) Return zero-filled state tensor(s).
Args:
batch_size: int, float, or unit Tensor representing the batch size.
dtype: the data type to use for the state.
End of explanation
with tf.variable_scope('rnnlm',reuse=False):
    softmax_w = tf.get_variable("softmax_w", [rnn_size, vocab_size]) #128x65
    softmax_b = tf.get_variable("softmax_b", [vocab_size]) # 1x65)
    with tf.device("/cpu:0"):
        embedding = tf.get_variable("embedding", [vocab_size, rnn_size])  #65x128
        #input_data is a matrix of 60x50 and embedding is dictionary of 65x128 for all 65 characters
        # embedding_lookup goes to each row of input_data, and for each character in the row, finds the correspond vector in embedding
        # it creates a 60*50*[1*128] matrix
        # so, the first elemnt of em, is a matrix of 50x128, which each row of it is vector representing that character
        em = tf.nn.embedding_lookup(embedding, input_data) # em is 60x50x[1*128]
        # split: Splits a tensor into sub tensors.
        # syntax:  tf.split(split_dim, num_split, value, name='split')
        # it will split the 60x50x[1x128] matrix into 50 matrix of 60x[1*128]
        inputs = tf.split(em, seq_length, 1)
        # It will convert the list to 50 matrix of [60x128]
        inputs = [tf.squeeze(input_, [1]) for input_ in inputs]
session.run(tf.global_variables_initializer())
session.run(embedding)
em = tf.nn.embedding_lookup(embedding, input_data)
em
emp = session.run(em,feed_dict={input_data:x})
emp.shape
emp[0]
inputs = tf.split(em, seq_length, 1)
inputs[0:5]
inputs = [tf.squeeze(input_, [1]) for input_ in inputs]
inputs[0:5]
Explanation: Embedding
End of explanation
session.run(inputs[0],feed_dict={input_data:x})
stacked_cell.state_size
#outputs is 50x[60*128]
outputs, last_state = tf.contrib.legacy_seq2seq.rnn_decoder(inputs, initial_state, stacked_cell, loop_function=None, scope='rnnlm')
outputs[0:5]
test = outputs[0]
test
session.run(tf.global_variables_initializer())
session.run(test,feed_dict={input_data:x})
Explanation: Feeding a batch of 50 sequence to a RNN:
Step 1:  first character of each of the 50 sentences (in a batch) is input in parallel.  
Step 2:  second character of each of the 50 sentences is input in parallel. 
Step n: nth character of each of the 50 sentences is input in parallel.  
The parallelism is only for efficiency.  Each character in a batch is handled in parallel,  but the network sees one character of a sequence at a time and does the computations accordingly. All the computations involving the characters of all sequences in a batch at a given time step are done in parallel.
End of explanation
output = tf.reshape(tf.concat(outputs, 1), [-1, rnn_size])
output
logits = tf.matmul(output, softmax_w) + softmax_b
logits
probs = tf.nn.softmax(logits)
probs
session.run(tf.global_variables_initializer())
session.run(probs,feed_dict={input_data:x})
loss = tf.contrib.legacy_seq2seq.sequence_loss_by_example([logits],
                [tf.reshape(targets, [-1])],
                [tf.ones([batch_size * seq_length])],
                vocab_size)
cost = tf.reduce_sum(loss) / batch_size / seq_length
cost
        
final_state = last_state
final_state
lr = tf.Variable(0.0, trainable=False)
grad_clip =5.
tvars = tf.trainable_variables()
tvars
session.run(tf.global_variables_initializer())
[v.name for v in tf.global_variables()]
grads, _ = tf.clip_by_global_norm(tf.gradients(cost, tvars), grad_clip)
grads
session.run(grads, feed_dict)[0]
optimizer = tf.train.AdamOptimizer(lr)
train_op = optimizer.apply_gradients(zip(grads, tvars))
Explanation: outputs is 50x[60*128]. We need to reshape it to [60x50x128]. Then we can calculate the softmax:
softmax_w is [rnn_size, vocab_size], [128x65]
[60x50x128]x[128x65]+[60x50]
End of explanation
class LSTMModel():
    def __init__(self,sample=False):
        rnn_size = 128 # size of RNN hidden state vector
        batch_size = 60 # minibatch size, i.e. size of dataset in each epoch
        seq_length = 50 # RNN sequence length
        num_layers = 2 # number of layers in the RNN
        vocab_size = 65
        grad_clip = 5.
        if sample:
            print("sample mode")
            batch_size = 1
            seq_length = 1
        # model.cell.state_size is (128, 128)
        with tf.variable_scope('lstm_model_cell'):
            reuse = tf.get_variable_scope().reuse
            self.stacked_cell = tf.contrib.rnn.MultiRNNCell([tf.contrib.rnn.BasicRNNCell(rnn_size, reuse=reuse) 
                                                         for _ in range(num_layers)])
        self.input_data = tf.placeholder(tf.int32, [batch_size, seq_length])
        self.targets = tf.placeholder(tf.int32, [batch_size, seq_length])
        # Initial state of the LSTM memory.
        # The memory state of the network is initialized with a vector of zeros and gets updated after reading each char. 
        self.initial_state = stacked_cell.zero_state(batch_size, tf.float32) #why batch_size
        with tf.variable_scope('rnnlm_class1'):
            softmax_w = tf.get_variable("softmax_w", [rnn_size, vocab_size]) #128x65
            softmax_b = tf.get_variable("softmax_b", [vocab_size]) # 1x65
            with tf.device("/cpu:0"):
                embedding = tf.get_variable("embedding", [vocab_size, rnn_size])  #65x128
                inputs = tf.split(tf.nn.embedding_lookup(embedding, self.input_data), seq_length, 1)
                inputs = [tf.squeeze(input_, [1]) for input_ in inputs]
                #inputs = tf.split(em, seq_length, 1)
        # The value of state is updated after processing each batch of chars.
        outputs, last_state = tf.contrib.legacy_seq2seq.rnn_decoder(inputs, self.initial_state, self.stacked_cell, loop_function=None, scope='rnnlm_class1')
        output = tf.reshape(tf.concat(outputs,1), [-1, rnn_size])
        self.logits = tf.matmul(output, softmax_w) + softmax_b
        self.probs = tf.nn.softmax(self.logits)
        loss = tf.contrib.legacy_seq2seq.sequence_loss_by_example([self.logits],
                [tf.reshape(self.targets, [-1])],
                [tf.ones([batch_size * seq_length])],
                vocab_size)
        self.cost = tf.reduce_sum(loss) / batch_size / seq_length
        self.final_state = last_state
        self.lr = tf.Variable(0.0, trainable=False)
        tvars = tf.trainable_variables()
        grads, _ = tf.clip_by_global_norm(tf.gradients(self.cost, tvars),grad_clip)
        optimizer = tf.train.AdamOptimizer(self.lr)
        self.train_op = optimizer.apply_gradients(zip(grads, tvars))
        
    def sample(self, sess, chars, vocab, num=200, prime='The ', sampling_type=1):
        state = sess.run(self.stacked_cell.zero_state(1, tf.float32))
        for char in prime[:-1]:
            x = np.zeros((1, 1))
            x[0, 0] = vocab[char]
            feed = {self.input_data: x, self.initial_state:state}
            [state] = sess.run([self.final_state], feed)
        def weighted_pick(weights):
            t = np.cumsum(weights)
            s = np.sum(weights)
            return(int(np.searchsorted(t, np.random.rand(1)*s)))
        ret = prime
        char = prime[-1]
        for n in range(num):
            x = np.zeros((1, 1))
            x[0, 0] = vocab[char]
            feed = {self.input_data: x, self.initial_state:state}
            [probs, state] = sess.run([self.probs, self.final_state], feed)
            p = probs[0]
            if sampling_type == 0:
                sample = np.argmax(p)
            elif sampling_type == 2:
                if char == ' ':
                    sample = weighted_pick(p)
                else:
                    sample = np.argmax(p)
            else: # sampling_type == 1 default:
                sample = weighted_pick(p)
            pred = chars[sample]
            ret += pred
            char = pred
        return ret
Explanation: Using classes
Now that we have learned how the networks work, we can put all together:
End of explanation
with tf.variable_scope("rnn"):
    model = LSTMModel()
sess = tf.InteractiveSession()
sess.run(tf.global_variables_initializer())
e=1
sess.run(tf.assign(model.lr, learning_rate * (decay_rate ** e)))
data_loader.reset_batch_pointer()
state = sess.run(model.initial_state)
state
x, y = data_loader.next_batch()
feed = {model.input_data: x, model.targets: y, model.initial_state:state}
train_loss, state, _ = sess.run([model.cost, model.final_state, model.train_op], feed)
train_loss
state
Explanation: the input is always a matrix of of shape [n x m]. Where n is the batch size, m is the feature size. 
In our case, the input shape will be [60 x ??]. 
size of data is 1113000, number of batches are 371, batch size is 60 and sequence length is 50. so, 5060371= 1113000
we have 50 epochs. 
each input matrix will represent 1 update per epoch.
Creating the LSTM object
End of explanation
initial_lr = 0.01
num_epochs = 50
with tf.Session() as sess:
    sess.run(tf.global_variables_initializer())
    for e in range(num_epochs): # num_epochs is 5 for test, but should be higher
        current_lr = initial_lr * (decay_rate ** e)
        sess.run(tf.assign(model.lr, current_lr))
        print('Epoch {} ({} / {} batches, lr={:.4f})'.format(
            e+1,
            (e+1) * data_loader.num_batches, 
            num_epochs * data_loader.num_batches,
            current_lr
        ))
        data_loader.reset_batch_pointer()
        state = sess.run(model.initial_state) # (2x[60x128])
        for b in range(data_loader.num_batches): #for each batch
            start = time.time()
            x, y = data_loader.next_batch()
            feed = {model.input_data: x, model.targets: y, model.initial_state:state}
            train_loss, state, _ = sess.run([model.cost, model.final_state, model.train_op], feed)
            end = time.time()
        print("Train_loss={:.3f}   Time/Batch={:.3f} ms".format(
            train_loss, 
            (end - start) * 1000
        ))
        print()
        #model.sample(sess, data_loader.chars , data_loader.vocab, num=200, prime='The ', sampling_type=1)
Explanation: Train usinng LSTMModel class
End of explanation
sess = tf.InteractiveSession()
with tf.variable_scope("sample_test"):
    sess.run(tf.global_variables_initializer())
    m = LSTMModel(sample=True)
prime='The '
num=200
sampling_type=1
vocab=data_loader.vocab
chars=data_loader.chars 
sess.run(m.initial_state)
#print state
sess.run(tf.global_variables_initializer())
state=sess.run(m.initial_state)
for char in prime[:-1]:
    x = np.zeros((1, 1))
    x[0, 0] = vocab[char]
    feed = {m.input_data: x, m.initial_state:state}
    [state] = sess.run([m.final_state], feed)
state
def weighted_pick(weights):
    t = np.cumsum(weights)
    s = np.sum(weights)
    return(int(np.searchsorted(t, np.random.rand(1)*s)))
ret = prime
char = prime[-1]
for n in range(num):
    x = np.zeros((1, 1))
    x[0, 0] = vocab[char]
    feed = {m.input_data: x, m.initial_state:state}
    [probs, state] = sess.run([m.probs, m.final_state], feed)
    p = probs[0]
    if sampling_type == 0:
        sample = np.argmax(p)
    elif sampling_type == 2:
        if char == ' ':
            sample = weighted_pick(p)
        else:
            sample = np.argmax(p)
    else: # sampling_type == 1 default:
        sample = weighted_pick(p)
    pred = chars[sample]
    ret += pred
    char = pred
ret
Explanation: Sample
End of explanation
sess = tf.InteractiveSession()
sess.run(tf.global_variables_initializer())
state=sess.run(m.initial_state)
m.sample(sess, data_loader.chars , data_loader.vocab, num=200, prime='The ', sampling_type=1)
Explanation: Sample using function
End of explanation
<END_TASK>