<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 Wayne Nixalo - 1 Jul 2017
RNN practice in Theano -- 4th attempt
Implementing dimensionality for context idea.
Theano RNN
Step1:  So here's the trouble spot. First looking at the shape & visualizing xs, then seeing how we can get xs to be of the right shape
Step2:  By trial & error found out axis=2 is the right axis to expand. And looks like it gives me exactly what I was looking for.
Step3:  Now to build an RNN in Theano and test it out. A successful result is a loss of around ~14.4, and an okay'ish prediction of the next character in an 8 character series.
Step4:  It's a little better but no where near good enough. I need to find out what's wrong..
Step5:  xs3 is in the proper format.. so I have to look at the oh_xs3 and oh_xs3_rnn versions
<ASSISTANT_TASK:>
Python Code:
import theano
import os, sys
sys.path.insert(1, os.path.join('../utils'))
from utils import *
# from __future__ import division, print_functions
path = get_file('nietzsche.txt', origin="https://s3.amazonaws.com/text-datasets/nietzsche.txt")
text = open(path).read()
print('corpus length:', len(text))
chars = sorted(list(set(text)))
vocab_size = len(chars) + 1
print('total chars:', vocab_size)
chars.insert(0, "\0")
''.join(chars[1:-6])
char_indices = dict((c, i) for i, c in enumerate(chars))
indices_char = dict((i, c) for i, c in enumerate(chars))
idx = [char_indices[c] for c in text]
# ''.join(indices_char[i] for i in idx[:70])
n_hidden, n_fac, cs = 256, 42, 8
c_in_dat = [[idx[i+n] for i in xrange(0, len(idx)-1-cs, cs)] for n in range(cs)]
c_out_dat = [[idx[i+cs] for i in xrange(0, len(idx)-cs, cs)] for n in range(cs)]
xs = [np.stack(c[:-2]) for c in c_in_dat]
ys = [np.stack(c[:-2]) for c in c_out_dat]
Explanation: Wayne Nixalo - 1 Jul 2017
RNN practice in Theano -- 4th attempt
Implementing dimensionality for context idea.
Theano RNN
End of explanation
for row in [xs[n][:cs] for n in range(cs)]: print row
xs[0].shape
Explanation: So here's the trouble spot. First looking at the shape & visualizing xs, then seeing how we can get xs to be of the right shape
End of explanation
# xs2 = np.stack(np.squeeze(xs), axis=0)
xs2 = np.expand_dims(xs, axis=2)
xs2[0].shape
for row in [xs2[n][:cs] for n in range(cs)]: print row
xs2[0].shape
print(len(xs), len(xs2))
Explanation: By trial & error found out axis=2 is the right axis to expand. And looks like it gives me exactly what I was looking for.
End of explanation
# One Hot Encoding xs & ys:
oh_ys = [to_categorical(o, vocab_size) for o in ys]
oh_y_rnn = np.stack(oh_ys, axis=1)
# seeing if there's a difference between the x & x2 versions
oh_xs = [to_categorical(o, vocab_size) for o in xs]
oh_x_rnn = np.stack(oh_xs, axis=1)
oh_xs2 = [to_categorical(o, vocab_size) for o in xs2]
oh_x2_rnn = np.stack(oh_xs, axis=1)
xs3 = [np.stack(x[:]) for x in xs2]
oh_xs3 = [to_categorical(o, vocab_size) for o in xs3]
oh_x3_rnn = np.stack(oh_xs, axis=1)
oh_x_rnn.shape, oh_x2_rnn.shape, oh_x3_rnn.shape, oh_y_rnn.shape
n_input = vocab_size
n_output = vocab_size
def init_wgts(rows, cols):
    scale = math.sqrt(2/rows)
    return shared(normal(scale=scale, size=(rows,cols)).astype(np.float32))
def init_bias(rows):
    return shared(np.zeros(rows, dtype=np.float32))
def wgts_and_bias(n_in, n_out):
    return init_wgts(n_in, n_out), init_bias(n_out)
def id_and_bias(n):
    return shared(np.eye(n, dtype=np.float32)), init_bias(n)
# Theano Variables
t_inp = T.matrix('inp')
t_outp = T.matrix('outp')
t_h0 = T.vector('h0')
lr = T.scalar('lr')
all_args = [t_h0, t_inp, t_outp, lr]
W_h = id_and_bias(n_hidden)
W_x = wgts_and_bias(n_input, n_hidden)
W_y = wgts_and_bias(n_hidden, n_output)
w_all = list(chain.from_iterable([W_h, W_x, W_y]))
def step(x, h, W_h, b_h, W_x, b_x, W_y, b_y):
    # Calculate the hidden activations
    h = nnet.relu(T.dot(x, W_x) + b_x + T.dot(h, W_h) + b_h)
    # Calculate the output activations
    y = nnet.softmax(T.dot(h, W_y) + b_y)
    # Return both (the `Flatten()` is to work around a Theano bug)
    return h, T.flatten(y, 1)
[v_h, v_y], _ = theano.scan(step, sequences=t_inp,
                             outputs_info=[t_h0, None], non_sequences=w_all)
error = nnet.categorical_crossentropy(v_y, t_outp).sum()
g_all = T.grad(error, w_all)
def upd_dict(wgts, grads, lr):
    return OrderedDict({w: w - g * lr for (w,g) in zip(wgts, grads)})
upd = upd_dict(w_all, g_all, lr)
# finally ready to compile the function
fn = theano.function(all_args, error, updates=upd, allow_input_downcast=True)
X = oh_x_rnn
X2 = oh_x2_rnn
X3 = oh_x3_rnn
Y = oh_y_rnn
X.shape, X2.shape, X3.shape, Y.shape
# gonna run on X and X2 and X3 simultaneously 
err = 0.0; err2 = 0.0; err3 = 0.0; l_rate = 0.01
for i in xrange(len(X)):
    err += fn(np.zeros(n_hidden), X[i], Y[i], l_rate)
    err2 += fn(np.zeros(n_hidden), X2[i], Y[i], l_rate)
    err3 += fn(np.zeros(n_hidden), X3[i], Y[i], l_rate)
    if i % 1000 == 999:
        print ("ErrorX:{:.3f}   ErrorX2:{:.3f}   ErrorX2:{:.3f}".format(err/1000, err2/1000, err3/1000))
        err=0.0; err2=0.0; err3=0.0
Explanation: Now to build an RNN in Theano and test it out. A successful result is a loss of around ~14.4, and an okay'ish prediction of the next character in an 8 character series.
End of explanation
xs3 = [np.stack(x[:]) for x in xs2]
xs3[0].shape
xs3
Explanation: It's a little better but no where near good enough. I need to find out what's wrong..
End of explanation
oh_xs3[0]
Explanation: xs3 is in the proper format.. so I have to look at the oh_xs3 and oh_xs3_rnn versions
End of explanation
<END_TASK>