<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 Recurrent postprocessing
Step1:  Gather data
Step2:  Network
Step3:  Train the model
Step4:  Analyze training
<ASSISTANT_TASK:>
Python Code:
#Create references to important directories we will use over and over
import os, sys
DATA_HOME_DIR = '/home/nathan/olin/spring2017/line-follower/line-follower/data'
#import modules
import numpy as np
from glob import glob
from PIL import Image
from tqdm import tqdm
import bcolz
from matplotlib import pyplot as plt
import seaborn as sns
%matplotlib inline
from keras.layers import *
from keras.layers.recurrent import LSTM
from keras.models import Sequential
from keras.optimizers import Adam
from keras.layers.wrappers import TimeDistributed
from keras.metrics import categorical_crossentropy, categorical_accuracy
%cd $DATA_HOME_DIR
path = DATA_HOME_DIR
train_path1=path + '/sun_apr_16_office_full_line_1'
train_path2=path + '/qea_blob_1'
# valid_path1=path + '/sun_apr_16_office_full_line_2'
# valid_path2=path + '/qea_blob_2'
valid_path1=path + '/qea-square_3'#+ '/sun_apr_16_office_full_line_2'
# train_path=path + '/qea_blob_1'
# valid_path=path + '/qea_blob_2'
Explanation: Recurrent postprocessing: v1
This file takes as input a sequence of data from the CNN, and attempts to refine that into a more accurate output command.
Some credit belongs to https://github.com/harvitronix/five-video-classification-methods/blob/master/models.py for providing inspiration.
End of explanation
INPUT_LEN = 512 # The number of columns in the CSV
WINDOW_SIZE = 16
def load_array(fname):
    return bcolz.open(fname)[:]
def windows(X, Y, seq_len=10):
    assert len(X) == len(Y)
    
    result = []
    for index in range(X.shape[0] - seq_len):
        result.append([X[index:index+seq_len], Y[index+seq_len-1]])
    result = np.array(result)
#     np.random.shuffle(result)
    return np.array(list(result[:,0])), np.array(list(result[:,1]))
def get_data(paths):
    Y_return = []
    for path in paths:
        %cd $path
        Y_train = np.genfromtxt('cmd_vel.csv', delimiter=',')[:,1] # only use turning angle
        Y_train = np.concatenate((Y_train, Y_train*-1))
        
        Y_return.extend(Y_train)
        
    X_all = load_array(paths[-1]+'/X_train_features3.b')
    X_all = np.reshape(X_all, (len(X_all), INPUT_LEN))
        
    print (len(X_all), len(Y_return))
    
    X_windowed, Y_windowed = windows(X_all, Y_return, WINDOW_SIZE)
    return np.array(X_windowed), np.array(Y_windowed)
X_train, Y_train = get_data([train_path1, train_path2])
X_valid, Y_valid = get_data([valid_path1])
X_train.shape
Explanation: Gather data
End of explanation
in_shape = (WINDOW_SIZE, INPUT_LEN)
def get_model():
    model = Sequential([
            LSTM(2048, return_sequences=False, input_shape=in_shape),
            Dropout(0.5),
#             Flatten(input_shape=in_shape),
            Dense(512, activation='relu'),
#             Dense(512, activation='relu'),
#             Dense(512, activation='relu'),
            Dropout(0.5),
            Dense(1)
        ])
    model.compile(loss='mean_absolute_error', optimizer='adam')
    
    return model
    
model = get_model()
model.summary()
Explanation: Network
End of explanation
%cd $DATA_HOME_DIR
model.load_weights('LSTM_postprocessor_v1.h5')
history = model.fit(X_train, Y_train,#X_train[:,-1],
                    batch_size = 96,
                    nb_epoch=150,
                    validation_data=(X_valid, Y_valid),
                    verbose=True)
conv_predictions = X_train[:,-1]
recurrent_predictions = model.predict(X_train)
ground_truth = Y_train
for x,y,z in zip(conv_predictions[:,0], ground_truth, recurrent_predictions[:,0]):
    print ("{:07f}\t{:07f}\t{:07f}\t".format(x,y,z))
conv_predictions = X_valid[:,-1]
recurrent_predictions = model.predict(X_valid)
ground_truth = Y_valid
for x,y,z in zip(conv_predictions[:,0], ground_truth, recurrent_predictions[:,0]):
    print ("{:07f}\t{:07f}\t{:07f}\t".format(x,y,z))
plt.plot(ground_truth[:100])
Explanation: Train the model
End of explanation
sns.tsplot(history.history['val_loss'])
for x,y in zip(X_valid, Y_valid):
    print ("{}\t{}".format(np.reshape(x,(10,)),y))
Explanation: Analyze training
End of explanation
<END_TASK>