<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 Setup
We're going to download the collected works of Nietzsche to use as our data for this class.
Step1:  Sometimes it's useful to have a zero value in the dataset, e.g. for padding
Step2:  Map from chars to indices and back again
Step3:  idx will be the data we use from now own - it simply converts all the characters to their index (based on the mapping above)
Step4:  3 char model
Create inputs
Create a list of every 4th character, starting at the 0th, 1st, 2nd, then 3rd characters
Step5:  Our inputs
Step6:  Our output
Step7:  The first 4 inputs and outputs
Step8:  The number of latent factors to create (i.e. the size of the embedding matrix)
Step9:  Create inputs and embedding outputs for each of our 3 character inputs
Step10:  Create and train model
Pick a size for our hidden state
Step11:  This is the 'green arrow' from our diagram - the layer operation from input to hidden.
Step12:  Our first hidden activation is simply this function applied to the result of the embedding of the first character.
Step13:  This is the 'orange arrow' from our diagram - the layer operation from hidden to hidden.
Step14:  Our second and third hidden activations sum up the previous hidden state (after applying dense_hidden) to the new input state.
Step15:  This is the 'blue arrow' from our diagram - the layer operation from hidden to output.
Step16:  The third hidden state is the input to our output layer.
Step17:  Test model
Step18:  RNNs
Create inputs
This is the size of our unrolled RNN.
Step19:  For each of 0 through 7, create a list of every 8th character with that starting point. These will be the 8 inputs to out model.
Step20:  Then create a list of the next character in each of these series. This will be the labels for our model.
Step21:  So each column below is one series of 8 characters from the text.
Step22:  ...and this is the next character after each sequence.
Step23:  Our first RNN!
Create and train model
Step24:  The first character of each sequence goes through dense_in(), to create our first hidden activations.
Step25:  Then for each successive layer we combine the output of dense_in() on the next character with the output of dense_hidden() on the current hidden state, to create the new hidden state.
Step26:  Putting the final hidden state through dense_out() gives us our output.
Step27:  So now we can create our model.
Step28:  Test model
Step29:  Our first RNN with keras!
Step30:  This is nearly exactly equivalent to the RNN we built ourselves in the previous section.
Step31:  Returning sequences
Create inputs
To use a sequence model, we can leave our input unchanged - but we have to change our output to a sequence (of course!)
Here, c_out_dat is identical to c_in_dat, but moved across 1 character.
Step32:  Reading down each column shows one set of inputs and outputs.
Step33:  Create and train model
Step34:  We're going to pass a vector of all zeros as our starting point - here's our input layers for that
Step35:  Test model
Step36:  Sequence model with keras
Step37:  To convert our previous keras model into a sequence model, simply add the 'return_sequences=True' parameter, and add TimeDistributed() around our dense layer.
Step38:  One-hot sequence model with keras
This is the keras version of the theano model that we're about to create.
Step39:  Stateful model with keras
Step40:  A stateful model is easy to create (just add "stateful=True") but harder to train. We had to add batchnorm and use LSTM to get reasonable results.
When using stateful in keras, you have to also add 'batch_input_shape' to the first layer, and fix the batch size there.
Step41:  Since we're using a fixed batch shape, we have to ensure our inputs and outputs are a even multiple of the batch size.
Step42:  Keras GRU
Identical to the last keras rnn, but a GRU!
<ASSISTANT_TASK:>
Python Code:
path = get_file('nietzsche.txt', origin="https://s3.amazonaws.com/text-datasets/nietzsche.txt")
text = open(path).read()
print('corpus length:', len(text))
chars = sorted(list(set(text)))
vocab_size = len(chars)+1
print('total chars:', vocab_size)
Explanation: Setup
We're going to download the collected works of Nietzsche to use as our data for this class.
End of explanation
chars.insert(0, "\0")
''.join(chars[1:-5])
Explanation: Sometimes it's useful to have a zero value in the dataset, e.g. for padding
End of explanation
char_indices = dict((c, i) for i, c in enumerate(chars))
indices_char = dict((i, c) for i, c in enumerate(chars))
char_indices;
Explanation: Map from chars to indices and back again
End of explanation
idx = [char_indices[c] for c in text]
idx[:10]
''.join(indices_char[i] for i in idx[:70])
Explanation: idx will be the data we use from now own - it simply converts all the characters to their index (based on the mapping above)
End of explanation
cs=3
c1_dat = [idx[i] for i in range(0, len(idx)-1-cs, cs)]
c2_dat = [idx[i+1] for i in range(0, len(idx)-1-cs, cs)]
c3_dat = [idx[i+2] for i in range(0, len(idx)-1-cs, cs)]
c4_dat = [idx[i+3] for i in range(0, len(idx)-1-cs, cs)]
c1_dat[:10]
Explanation: 3 char model
Create inputs
Create a list of every 4th character, starting at the 0th, 1st, 2nd, then 3rd characters
End of explanation
# just creates an array out of list
xxx = np.stack(c1_dat[:-2])
xxx.shape
xxx[:10]
x1 = np.stack(c1_dat[:-2])
x2 = np.stack(c2_dat[:-2])
x3 = np.stack(c3_dat[:-2])
Explanation: Our inputs
End of explanation
y = np.stack(c4_dat[:-2])
Explanation: Our output
End of explanation
x1[:5], x2[:4], x3[:4]
y[:4]
x1.shape, y.shape
Explanation: The first 4 inputs and outputs
End of explanation
n_fac = 42
Explanation: The number of latent factors to create (i.e. the size of the embedding matrix)
End of explanation
def embedding_input(name, n_in, n_out):
    inp = Input(shape=(1,), dtype='int64', name=name)
    emb = Embedding(n_in, n_out, input_length=1)(inp)
    return inp, Flatten()(emb)
c1_in, c1 = embedding_input('c1', vocab_size, n_fac)
c2_in, c2 = embedding_input('c2', vocab_size, n_fac)
c3_in, c3 = embedding_input('c3', vocab_size, n_fac)
Explanation: Create inputs and embedding outputs for each of our 3 character inputs
End of explanation
n_hidden = 256
Explanation: Create and train model
Pick a size for our hidden state
End of explanation
dense_in = Dense(n_hidden, activation='relu')
Explanation: This is the 'green arrow' from our diagram - the layer operation from input to hidden.
End of explanation
c1_hidden = dense_in(c1)
Explanation: Our first hidden activation is simply this function applied to the result of the embedding of the first character.
End of explanation
dense_hidden = Dense(n_hidden, activation='tanh')
Explanation: This is the 'orange arrow' from our diagram - the layer operation from hidden to hidden.
End of explanation
c2_dense = dense_in(c2)
hidden_2 = dense_hidden(c1_hidden)
c2_hidden = merge([c2_dense, hidden_2])
c3_dense = dense_in(c3)
hidden_3 = dense_hidden(c2_hidden)
c3_hidden = merge([c3_dense, hidden_3])
Explanation: Our second and third hidden activations sum up the previous hidden state (after applying dense_hidden) to the new input state.
End of explanation
dense_out = Dense(vocab_size, activation='softmax')
Explanation: This is the 'blue arrow' from our diagram - the layer operation from hidden to output.
End of explanation
c4_out = dense_out(c3_hidden)
model = Model([c1_in, c2_in, c3_in], c4_out)
model.compile(loss='sparse_categorical_crossentropy', optimizer=Adam())
model.optimizer.lr=0.000001
model.fit([x1, x2, x3], y, batch_size=64, nb_epoch=4)
model.optimizer.lr=0.01
model.fit([x1, x2, x3], y, batch_size=64, nb_epoch=4)
model.optimizer.lr=0.000001
model.fit([x1, x2, x3], y, batch_size=64, nb_epoch=4)
model.optimizer.lr=(0.01)
model.fit([x1, x2, x3], y, batch_size=64, nb_epoch=4)
Explanation: The third hidden state is the input to our output layer.
End of explanation
def get_next(inp):
    idxs = [char_indices[c] for c in inp]
    print(idxs)
    arrs = [np.array(i)[np.newaxis] for i in idxs]
    print(arrs)
    p = model.predict(arrs)
    i = np.argmax(p)
    print(p, i)
    return chars[i]
get_next('phi')
get_next(' th')
get_next(' an')
Explanation: Test model
End of explanation
cs=8
Explanation: RNNs
Create inputs
This is the size of our unrolled RNN.
End of explanation
l = len(idx)-cs-1  # relevant for later sequences
c_in_dat = [[idx[i+n] for i in range(0, l, cs)]
            for n in range(cs)]
# dimensions
len(idx), l/cs
[len(c_in_dat[i]) for i in range(cs)],  len(c_in_dat)
# python does not support multi-axes slicing in lists
c_in_dat[0][0:10]
c_in_dat[7][0:10]
Explanation: For each of 0 through 7, create a list of every 8th character with that starting point. These will be the 8 inputs to out model.
End of explanation
# same as c_out_dat[0] shifted by 1
c_out_dat = [idx[i+cs] for i in range(0, l, cs)]
c_out_dat[0:10]
xs = [np.stack(c[:-2]) for c in c_in_dat]
# transform into numpy array
len(xs), xs[0].shape
xs[0][0:10]
y = np.stack(c_out_dat[:-2])
Explanation: Then create a list of the next character in each of these series. This will be the labels for our model.
End of explanation
len(xs), xs[0].shape
[xs[n][:cs+10] for n in range(cs)]
Explanation: So each column below is one series of 8 characters from the text.
End of explanation
y[:cs+10]
n_fac = 42
Explanation: ...and this is the next character after each sequence.
End of explanation
def embedding_input(name, n_in, n_out):
    inp = Input(shape=(1,), dtype='int64', name=name+'_in')
    emb = Embedding(n_in, n_out, input_length=1, name=name+'_emb')(inp)
    return inp, Flatten()(emb)
c_ins = [embedding_input('c'+str(n), vocab_size, n_fac) for n in range(cs)]
c_ins
n_hidden = 256
dense_in = Dense(n_hidden, activation='relu') # green arrow
# init: no random init but identity matrix, so multiplication does not change the original,
# so hidden state does not change at all
# rnn hidden wheight matrix and relus: init with identity
dense_hidden = Dense(n_hidden, activation='relu', init='identity') # orange arrow
dense_out = Dense(vocab_size, activation='softmax') # blue arrow
Explanation: Our first RNN!
Create and train model
End of explanation
# get first character's embedding
hidden = dense_in(c_ins[0][1])
Explanation: The first character of each sequence goes through dense_in(), to create our first hidden activations.
End of explanation
for i in range(1,cs):
    c_dense = dense_in(c_ins[i][1])
    hidden = dense_hidden(hidden)
    hidden = merge([c_dense, hidden])
Explanation: Then for each successive layer we combine the output of dense_in() on the next character with the output of dense_hidden() on the current hidden state, to create the new hidden state.
End of explanation
c_out = dense_out(hidden)
Explanation: Putting the final hidden state through dense_out() gives us our output.
End of explanation
model = Model([c[0] for c in c_ins], c_out)
model.compile(loss='sparse_categorical_crossentropy', optimizer=Adam())
model.summary();
model.fit(xs, y, batch_size=64, nb_epoch=12)
Explanation: So now we can create our model.
End of explanation
def get_next(inp):
    idxs = [np.array(char_indices[c])[np.newaxis] for c in inp]
    p = model.predict(idxs)
    return chars[np.argmax(p)]
get_next('for thos')
get_next('part of ')
get_next('queens a')
Explanation: Test model
End of explanation
n_hidden, n_fac, cs, vocab_size = (256, 42, 8, 86)
Explanation: Our first RNN with keras!
End of explanation
model=Sequential([
        Embedding(vocab_size, n_fac, input_length=cs),
        SimpleRNN(n_hidden, activation='relu', inner_init='identity'),
        Dense(vocab_size, activation='softmax')
    ])
model.summary()
model.compile(loss='sparse_categorical_crossentropy', optimizer=Adam())
np.stack(xs,1).squeeze().shape
model.fit(np.stack(xs,1).squeeze(), y, batch_size=64, nb_epoch=8)
#model.fit(xs, y, batch_size=64, nb_epoch=8)
def get_next_keras(inp):
    idxs = [char_indices[c] for c in inp]
    arrs = np.array(idxs)[np.newaxis,:]
    p = model.predict(arrs)[0]
    return chars[np.argmax(p)]
get_next_keras('this is ')
get_next_keras('part of ')
get_next_keras('queens a')
Explanation: This is nearly exactly equivalent to the RNN we built ourselves in the previous section.
End of explanation
#c_in_dat = [[idx[i+n] for i in range(0, len(idx)-1-cs, cs)]
#            for n in range(cs)]
c_out_dat = [[idx[i+n] for i in range(1, len(idx)-cs, cs)]
            for n in range(cs)]
ys = [np.stack(c[:-2]) for c in c_out_dat]
Explanation: Returning sequences
Create inputs
To use a sequence model, we can leave our input unchanged - but we have to change our output to a sequence (of course!)
Here, c_out_dat is identical to c_in_dat, but moved across 1 character.
End of explanation
[xs[n][:cs] for n in range(cs)]
[ys[n][:cs] for n in range(cs)]
Explanation: Reading down each column shows one set of inputs and outputs.
End of explanation
dense_in = Dense(n_hidden, activation='relu')
dense_hidden = Dense(n_hidden, activation='relu', init='identity')
dense_out = Dense(vocab_size, activation='softmax', name='output')
Explanation: Create and train model
End of explanation
inp1 = Input(shape=(n_fac,), name='zeros')
hidden = dense_in(inp1)
outs = []
for i in range(cs):
    c_dense = dense_in(c_ins[i][1])
    hidden = dense_hidden(hidden)
    hidden = merge([c_dense, hidden], mode='sum')
    # every layer now has an output
    outs.append(dense_out(hidden))
model = Model([inp1] + [c[0] for c in c_ins], outs)
model.compile(loss='sparse_categorical_crossentropy', optimizer=Adam())
model.summary()
np.tile
zeros = np.tile(np.zeros(n_fac), (len(xs[0]),1))
zeros.shape
%%capture output
model.fit([zeros]+xs, ys, batch_size=64, nb_epoch=12)
output.show()
Explanation: We're going to pass a vector of all zeros as our starting point - here's our input layers for that:
End of explanation
def get_nexts(inp):
    idxs = [char_indices[c] for c in inp]
    arrs = [np.array(i)[np.newaxis] for i in idxs]
    p = model.predict([np.zeros(n_fac)[np.newaxis,:]] + arrs)
    print(list(inp))
    return [chars[np.argmax(o)] for o in p]
get_nexts(' this is')
get_nexts(' part of')
Explanation: Test model
End of explanation
n_hidden, n_fac, cs, vocab_size
Explanation: Sequence model with keras
End of explanation
model=Sequential([
        Embedding(vocab_size, n_fac, input_length=cs),
        SimpleRNN(n_hidden, return_sequences=True, activation='relu', inner_init='identity'),
        TimeDistributed(Dense(vocab_size, activation='softmax')),
    ])
model.summary()
model.compile(loss='sparse_categorical_crossentropy', optimizer=Adam())
xs[0].shape
len(ys)
x_rnn=np.stack(xs, axis=1)
#y_rnn=np.expand_dims(np.stack(ys, axis=1), -1)
y_rnn=np.stack(ys, axis=1)
x_rnn.shape, y_rnn.shape
x_rnn[0]
%%capture output
model.fit(x_rnn, y_rnn, batch_size=64, nb_epoch=8)
output.show()
def get_nexts_keras(inp):
    idxs = [char_indices[c] for c in inp]
    arr = np.array(idxs)[np.newaxis,:]
    p = model.predict(arr)[0]
    print(list(inp))
    return [chars[np.argmax(o)] for o in p]
get_nexts_keras(' this is')
Explanation: To convert our previous keras model into a sequence model, simply add the 'return_sequences=True' parameter, and add TimeDistributed() around our dense layer.
End of explanation
model=Sequential([
        SimpleRNN(n_hidden, return_sequences=True, input_shape=(cs, vocab_size),
                  activation='relu', inner_init='identity'),
        TimeDistributed(Dense(vocab_size, activation='softmax')),
    ])
model.compile(loss='categorical_crossentropy', optimizer=Adam())
oh_ys = [to_categorical(o, vocab_size) for o in ys]
oh_y_rnn=np.stack(oh_ys, axis=1)
oh_xs = [to_categorical(o, vocab_size) for o in xs]
oh_x_rnn=np.stack(oh_xs, axis=1)
oh_x_rnn.shape, oh_y_rnn.shape
%%capture output
model.fit(oh_x_rnn, oh_y_rnn, batch_size=64, nb_epoch=8)
output.show()
def get_nexts_oh(inp):
    idxs = np.array([char_indices[c] for c in inp])
    arr = to_categorical(idxs, vocab_size)
    p = model.predict(arr[np.newaxis,:])[0]
    print(list(inp))
    return [chars[np.argmax(o)] for o in p]
get_nexts_oh(' this is')
Explanation: One-hot sequence model with keras
This is the keras version of the theano model that we're about to create.
End of explanation
bs=64
Explanation: Stateful model with keras
End of explanation
model=Sequential([
        Embedding(vocab_size, n_fac, input_length=cs, batch_input_shape=(bs,8)),
        BatchNormalization(),
        LSTM(n_hidden, return_sequences=True, stateful=True),
        TimeDistributed(Dense(vocab_size, activation='softmax')),
    ])
model.compile(loss='sparse_categorical_crossentropy', optimizer=Adam())
model.summary()
model.input_shape
Explanation: A stateful model is easy to create (just add "stateful=True") but harder to train. We had to add batchnorm and use LSTM to get reasonable results.
When using stateful in keras, you have to also add 'batch_input_shape' to the first layer, and fix the batch size there.
End of explanation
mx = len(x_rnn)//bs*bs
mx
%%capture output
model.fit(x_rnn[:mx], y_rnn[:mx], batch_size=bs, nb_epoch=4, shuffle=False)
output.show()
model.optimizer.lr=1e-4
%%capture output
model.fit(x_rnn[:mx], y_rnn[:mx], batch_size=bs, nb_epoch=4, shuffle=False)
output.show()
%%capture output
model.fit(x_rnn[:mx], y_rnn[:mx], batch_size=bs, nb_epoch=4, shuffle=False)
output.show()
Explanation: Since we're using a fixed batch shape, we have to ensure our inputs and outputs are a even multiple of the batch size.
End of explanation
model=Sequential([
        GRU(n_hidden, return_sequences=True, input_shape=(cs, vocab_size),
                  activation='relu', inner_init='identity'),
        TimeDistributed(Dense(vocab_size, activation='softmax')),
    ])
model.compile(loss='categorical_crossentropy', optimizer=Adam())
model.fit(oh_x_rnn, oh_y_rnn, batch_size=64, nb_epoch=8)
get_nexts_oh(' this is')
Explanation: Keras GRU
Identical to the last keras rnn, but a GRU!
End of explanation
<END_TASK>