<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 The histogram above shows the dataset is unbalanced.
We will now go ahead and drop most of the zero steering angles, which correspond to mostly straight driving.
Step1:  Now we start making changes to the driver log file and we create a new driver log file.
This new driver log file is created to later support larger datasets by adding more 'meta' columns with augmentation information. You'll notice that each augmentation method adds the columns it needs to later augment the image when loaded.
Step2:  We sample (normal dist) 40% of example that are not zero angles; flip them and reverse steering angle.
Step3:  We have to upsample under represented example and we do this by shifting images and their corresponding angles
Step4:  The histogram looks more balanced from the upsampling, but this caused us to created steering angles outside the bounds  of [-1,1]
The process_driver_log() method is used to pump out the final driver log file that we will use to generate the new driver log dataframe and also drops those examples.
Step5:  Now the histogram is mostly evenly distributed and the question now becomes how much data can we trim and still train the model?!
After some experimenting we can trim this histogram with 100 bins by trimming the bins with over 500 count.
The hacky solution I created below basically generates a list of strings that are panda dataframe masking code using the ranges from np.histogram() to search; then I loop through the list of code and run them through the eval() method to execute the code!
 The end result is no bins will go past 400 examples
Step6:  Now we have a decent starting point and we can start the pipeline process!!
The first thing we want to do is reduce the input shape size.
We do this by first cropping the top and bottom from the image that would be noise to the model. Then we resize the image to (64,64) and convert to HSV and only returning the S channel. This takes us from (160,320,3) to (64, 64, 1)!!
Step7:  Last step is to create the shift image and  load image methods.
Notice the load_image() method takes a dataframe row and uses the assumed meta columns to make the modification to the image on the fly. This assumes the preprocessing stage has added the columns needed.
Step8:  We can actually load everthing into memory and we don't need to use a generator!
Step16:  We are now ready to go into creating our model!
Main goals for us to think about while creating a model
Is it efficient for the task at hand?
If this was to be downloaded onto hardware in a car what would the power consumption and usuability be like?
There is an interesting paper called Deep Compression, which we don't implement here but it is food for thought and shows that model can be tiny and still work!
I started with a modified comma.ai model and I had a successful model, but it needed 300,000 trainable parameters. Reading the Deep compression paper and a blog post titled Self-driving car in a simulator with a tiny neural network I quickly realized that we can do better. The blog post linked shows a 63 parameter model that uses tiny images and a small network to get a stable model. I wanted to experiment and see if I could make it smaller and still stable. The solution developed over a number of experiments is a modified SqueezeNet implementation. With a squeeze net you get three addtional hyperparameters that are used to generate the fire module
Step18:  Here is the final 52 param model that we are using for this project.
<div align="left">
  </br>
  <img src="./images/SqueezeNet52.png"><br><br>
</div>
Through some experiments I know that once the loss reach around .03 we have a model that can drive in the simulator so we use a custom early termination callback to stop the training!
Step19:  To run the model training below you'll need to download and run the vis tool
Simply do the following
Step20:  Notice the loss doesn't change greatly and because we use an aggressive learning rate we could benefit from early termination.
This model was tiny (52 params!) and we are training it on a 2012 macbook air; looks like we could stop early and we could do this by writing our own keras callback to monitor and early terminate specifically for the val_loss ~0.3!
 Note
<ASSISTANT_TASK:>
Python Code:
zero_steering = drive_log_df[drive_log_df.steering == 0].sample(frac=0.9)
drive_log_df = drive_log_df.drop(zero_steering.index)
plt.figure(figsize=(10,4))
drive_log_df.steering.hist(bins=100, color='r')
plt.xlabel('steering angle bins')
plt.ylabel('counts')
plt.show()
print("Current Dataset Size: ", len(drive_log_df.steering))
Explanation: The histogram above shows the dataset is unbalanced.
We will now go ahead and drop most of the zero steering angles, which correspond to mostly straight driving.
End of explanation
def update_left_right_steering_correction(df):
    records = []
    for index, row in df.iterrows():
        left = row.left
        center = row.center
        right = row.right
        steering = row.steering
        
        records.append({
            'image': left,
            'steering': steering + 0.23
        })
        
        records.append({
            'image': right,
            'steering': steering - 0.23
        })
        
        records.append({
            'image': center,
            'steering': steering
        })
    
    return pd.DataFrame(data=records, columns=['image', 'steering'])
new_drive_log = update_left_right_steering_correction(drive_log_df)
new_drive_log.tail()
Explanation: Now we start making changes to the driver log file and we create a new driver log file.
This new driver log file is created to later support larger datasets by adding more 'meta' columns with augmentation information. You'll notice that each augmentation method adds the columns it needs to later augment the image when loaded.
End of explanation
def flip_images_augmentation(df):
    new_df = df[df.steering != 0].sample(frac=0.4)
    df.loc[:,'is_flipped'] = False
    new_df.loc[:,'is_flipped'] = True
    left_rows = (new_df.steering < 0)
    right_rows = (new_df.steering > 0)
    new_df.loc[left_rows,'steering'] = new_df[left_rows].steering.abs()
    new_df.loc[right_rows, 'steering'] = new_df[right_rows].steering * -1
    return pd.concat([df, new_df])
augmented = flip_images_augmentation(new_drive_log)
plt.figure(figsize=(10,4))
augmented.steering.hist(bins=100, color='r')
plt.xlabel('steering angle bins')
plt.ylabel('counts')
plt.show()
print("Current Dataset Size: ", len(augmented.steering))
Explanation: We sample (normal dist) 40% of example that are not zero angles; flip them and reverse steering angle.
End of explanation
def shift_img_augmentation(df):
    df.loc[:,'random_shift'] = 0
    new_df = df[df.steering != 0].copy()
    df.loc[:,'is_shift'] = False
    new_df.loc[:,'is_shift'] = True
    
    
    max_shift = 30
    max_ang = 0.17
    
    def row_shift_update(row):
        random_shift = np.random.randint(-max_shift, max_shift + 1)
        row.random_shift = random_shift
        updated_steer = row.steering + (random_shift / max_shift) * max_ang
        if abs(updated_steer) > 1:
            updated_steer = -1 if (updated_steer < 0) else 1
        row.steering = updated_steer
        return row
    new_df = new_df.apply(row_shift_update, axis=1)
    return pd.concat([df, new_df])
shifted = shift_img_augmentation(augmented)
plt.figure(figsize=(10,4))
shifted.steering.hist(bins=100, color='r')
plt.xlabel('steering angle bins')
plt.ylabel('counts')
plt.show()
print("Current Dataset Size: ", len(shifted.steering))
shifted.tail(1)
plt.show()
Explanation: We have to upsample under represented example and we do this by shifting images and their corresponding angles
End of explanation
def process_driver_log(driver_log):
    update_log = update_left_right_steering_correction(driver_log)
    update_log = flip_images_augmentation(update_log)
    update_log = shift_img_augmentation(update_log)
    #update_log = change_brightness_augmentation(update_log)
    
    #reset index since we it's no longer good.
    update_log = update_log.reset_index(drop=True)
    
    #drop outbound steering examples to be between [-1,1] !
    outbound_steering = update_log[abs(update_log.steering) > 1]
    update_log = update_log.drop(outbound_steering.index)
    return update_log
processed_log = process_driver_log(drive_log_df)
plt.figure(figsize=(10,4))
processed_log.steering.hist(bins=100, color='r')
plt.xlabel('steering angle bins')
plt.ylabel('counts')
plt.show()
print("Current Dataset Size: ", len(processed_log.steering))
Explanation: The histogram looks more balanced from the upsampling, but this caused us to created steering angles outside the bounds  of [-1,1]
The process_driver_log() method is used to pump out the final driver log file that we will use to generate the new driver log dataframe and also drops those examples.
End of explanation
hist, counts = np.histogram(processed_log.steering, bins=100)
upper_limit = 400
over = [(i, v) for i, v in enumerate(hist) if v > upper_limit ]
over_ranges = [(counts[i],counts[i+1]) for i,_ in over]
#loop through ranges and create a mask for each bin 
masks = ["processed_log[(processed_log.steering >= {0}) & (processed_log.steering < {1})]".format(l,r) for l,r in over_ranges]
for mask in masks:
    selected = eval(mask)
    selected_length = len(selected)
    frac_to_drop = (selected_length-upper_limit)/selected_length
    samples_to_drop = selected.sample(frac=frac_to_drop)
    processed_log = processed_log.drop(samples_to_drop.index)
plt.figure(figsize=(10,4))
processed_log.steering.hist(bins=100, color='g')
plt.xlabel('steering angle bins')
plt.ylabel('counts')
plt.show()
print("Current Dataset Size: ", len(processed_log.steering))
processed_log.to_csv('preprocessed_driver_log.csv')
Explanation: Now the histogram is mostly evenly distributed and the question now becomes how much data can we trim and still train the model?!
After some experimenting we can trim this histogram with 100 bins by trimming the bins with over 500 count.
The hacky solution I created below basically generates a list of strings that are panda dataframe masking code using the ranges from np.histogram() to search; then I loop through the list of code and run them through the eval() method to execute the code!
 The end result is no bins will go past 400 examples
End of explanation
def crop_top_and_bottom(image):
    resized = cv2.resize(image[70:140], (64,64),  cv2.INTER_AREA)
    return cv2.cvtColor(resized, cv2.COLOR_RGB2HSV)[:,:,1]
plt.figure(figsize=(10,4))
random_image = drive_log_df.iloc[0]
image = cv2.imread("./data/{0}".format(random_image.center))
print(image.shape)
plt.figure(1)
plt.imshow(image)
plt.show()
cropped_image = crop_top_and_bottom(image)
print(cropped_image.shape)
plt.figure(2)
plt.imshow(cropped_image)
plt.show()
Explanation: Now we have a decent starting point and we can start the pipeline process!!
The first thing we want to do is reduce the input shape size.
We do this by first cropping the top and bottom from the image that would be noise to the model. Then we resize the image to (64,64) and convert to HSV and only returning the S channel. This takes us from (160,320,3) to (64, 64, 1)!!
End of explanation
def shift_img(image, random_shift):
    rows, cols = image.shape
    mat = np.float32([[1, 0, random_shift], [0, 1, 0]])
    return cv2.warpAffine(image, mat, (cols, rows))
def load_image(row):
    image = cv2.imread("./data/{0}".format(row.image.strip()))
    image = cv2.cvtColor(image, cv2.COLOR_BGR2RGB)
    image = crop_top_and_bottom(image)
    
    if(row.is_flipped):
        image = cv2.flip(image,1)
    if(row.is_shift):
        image = shift_img(image, row.random_shift)  
    return image
Explanation: Last step is to create the shift image and  load image methods.
Notice the load_image() method takes a dataframe row and uses the assumed meta columns to make the modification to the image on the fly. This assumes the preprocessing stage has added the columns needed.
End of explanation
def load_all_features_and_labels(df):
    images = [load_image(row) for _, row in df.iterrows()]
    return np.array(images).reshape((len(images), 64, 64, 1)), df.steering 
features, labels = load_all_features_and_labels(processed_log)
print(features.shape, labels.shape)
Explanation: We can actually load everthing into memory and we don't need to use a generator!
End of explanation
def fire_module(x, fire_id, squeeze=16, expand=64):
    
    This is a modified version of: https://github.com/rcmalli/keras-squeezenet/blob/master/squeezenet.py#L14
    Changes made:
    * Uses ELU activation
    * Only supports tf 
    
    s_id = 'fire' + str(fire_id) + '/'
    c_axis = 3
    sq1x1 = "squeeze1x1"
    exp1x1 = "expand1x1"
    exp3x3 = "expand3x3"
    elu = "elu_"
  
    x = Convolution2D(squeeze, 1, 1, border_mode='valid', name=s_id + sq1x1)(x)
    x = Activation('elu', name=s_id + elu + sq1x1)(x)
    left = Convolution2D(expand, 1, 1, border_mode='valid', name=s_id + exp1x1)(x)
    left = Activation('elu', name=s_id + elu + exp1x1)(left)
    right = Convolution2D(expand, 3, 3, border_mode='same', name=s_id + exp3x3)(x)
    right = Activation('elu', name=s_id + elu + exp3x3)(right)
    x = merge([left, right], mode='concat', concat_axis=c_axis, name=s_id + 'concat')
    return x
def squeeze_model_10000():
    
    This model is a modification from the reference:
    https://github.com/rcmalli/keras-squeezenet/blob/master/squeezenet.py
    
    Normalizing will be done in the model directly for GPU speedup 
    
    input_shape=(64, 64, 1)
    input_img = Input(shape=input_shape)
    x = Lambda(lambda x: x/127.5 - 1.,input_shape=input_shape)(input_img)
    
    x = Convolution2D(2, 3, 3, subsample=(2, 2), border_mode='valid', name='conv1')(x)
    x = Activation('elu', name='elu_conv1')(x)
    x = MaxPooling2D(pool_size=(3, 3), strides=(2, 2), name='pool1')(x)
    x = fire_module(x, fire_id=2, squeeze=2, expand=6)
    x = fire_module(x, fire_id=3, squeeze=16, expand=64)
    x = Dropout(0.2, name='drop9')(x)
    x = GlobalAveragePooling2D()(x)
    out = Dense(1, name='loss')(x)
    model = Model(input=input_img, output=[out])
    plot(model, to_file='SqueezeNet10k.png', show_shapes=True)
    
    
    model.compile(optimizer=Adam(lr=1e-3), loss='mse')
    return model
def squeeze_model_1005():
    
    This model is a modification from the reference:
    https://github.com/rcmalli/keras-squeezenet/blob/master/squeezenet.py
    
    Normalizing will be done in the model directly for GPU speedup 
    
    input_shape=(64, 64, 1)
    input_img = Input(shape=input_shape)
    x = Lambda(lambda x: x/127.5 - 1.,input_shape=input_shape)(input_img)
    
    x = Convolution2D(2, 3, 3, subsample=(2, 2), border_mode='valid', name='conv1')(x)
    x = Activation('elu', name='elu_conv1')(x)
    x = MaxPooling2D(pool_size=(3, 3), strides=(2, 2), name='pool1')(x)
    x = fire_module(x, fire_id=2, squeeze=2, expand=6)
    x = fire_module(x, fire_id=3, squeeze=6, expand=12)
    x = Dropout(0.2, name='drop9')(x)
    x = GlobalAveragePooling2D()(x)
    out = Dense(1, name='loss')(x)
    model = Model(input=input_img, output=[out])
    plot(model, to_file='SqueezeNet1005.png', show_shapes=True)
    
    
    model.compile(optimizer=Adam(lr=1e-1), loss='mse')
    return model
def squeeze_model_329():
    
    This model is a modification from the reference:
    https://github.com/rcmalli/keras-squeezenet/blob/master/squeezenet.py
    
    Normalizing will be done in the model directly for GPU speedup 
    
    input_shape=(64, 64, 1)
    input_img = Input(shape=input_shape)
    x = Lambda(lambda x: x/127.5 - 1.,input_shape=input_shape)(input_img)
    
    x = Convolution2D(2, 3, 3, subsample=(2, 2), border_mode='valid', name='conv1')(x)
    x = Activation('elu', name='elu_conv1')(x)
    x = MaxPooling2D(pool_size=(3, 3), strides=(2, 2), name='pool1')(x)
    x = fire_module(x, fire_id=2, squeeze=2, expand=6)
    x = fire_module(x, fire_id=3, squeeze=2, expand=6)
    x = Dropout(0.2, name='drop9')(x)
    x = GlobalAveragePooling2D()(x)
    out = Dense(1, name='loss')(x)
    model = Model(input=input_img, output=[out])
    plot(model, to_file='SqueezeNet329.png', show_shapes=True)
    
    
    model.compile(optimizer=Adam(lr=1e-3), loss='mse')
    return model
def squeeze_model_159():
    
    This model is a modification from the reference:
    https://github.com/rcmalli/keras-squeezenet/blob/master/squeezenet.py
    
    Normalizing will be done in the model directly for GPU speedup 
    
    input_shape=(64, 64, 1)
    input_img = Input(shape=input_shape)
    x = Lambda(lambda x: x/127.5 - 1.,input_shape=input_shape)(input_img)
    
    x = Convolution2D(1, 3, 3, subsample=(2, 2), border_mode='valid', name='conv1')(x)
    x = Activation('elu', name='elu_conv1')(x)
    x = MaxPooling2D(pool_size=(3, 3), strides=(2, 2), name='pool1')(x)
    x = fire_module(x, fire_id=2, squeeze=2, expand=6)
    x = Dropout(0.2, name='drop9')(x)
    x = GlobalAveragePooling2D()(x)
    out = Dense(1, name='loss')(x)
    model = Model(input=input_img, output=[out])
    plot(model, to_file='SqueezeNet159.png', show_shapes=True)
    
    
    model.compile(optimizer=Adam(lr=1e-1), loss='mse')
    return model
def squeeze_model_63():
    
    This model is a modification from the reference:
    https://github.com/rcmalli/keras-squeezenet/blob/master/squeezenet.py
    
    Normalizing will be done in the model directly for GPU speedup 
    
    input_shape=(64, 64, 1)
    input_img = Input(shape=input_shape)
    x = Lambda(lambda x: x/127.5 - 1.,input_shape=input_shape)(input_img)
    
    x = Convolution2D(1, 3, 3, subsample=(2, 2), border_mode='valid', name='conv1')(x)
    x = Activation('elu', name='elu_conv1')(x)
    x = MaxPooling2D(pool_size=(3, 3), strides=(2, 2), name='pool1')(x)
    x = fire_module(x, fire_id=2, squeeze=2, expand=2)
    x = Dropout(0.2, name='drop9')(x)
    x = GlobalAveragePooling2D()(x)
    out = Dense(1, name='loss')(x)
    model = Model(input=input_img, output=[out])
    plot(model, to_file='SqueezeNet63.png', show_shapes=True)
    
    
    model.compile(optimizer=Adam(lr=1e-1), loss='mse')
    return model
def squeeze_model_52():
    
    This model is a modification from the reference:
    https://github.com/rcmalli/keras-squeezenet/blob/master/squeezenet.py
    
    Normalizing will be done in the model directly for GPU speedup 
    
    input_shape=(64, 64, 1)
    input_img = Input(shape=input_shape)
    x = Lambda(lambda x: x/127.5 - 1.,input_shape=input_shape)(input_img)
    
    x = Convolution2D(2, 3, 3, subsample=(2, 2), border_mode='valid', name='conv1')(x)
    x = Activation('elu', name='elu_conv1')(x)
    x = MaxPooling2D(pool_size=(3, 3), strides=(2, 2), name='pool1')(x)
    x = fire_module(x, fire_id=2, squeeze=1, expand=2)
    x = Dropout(0.2, name='drop3')(x)
    x = GlobalAveragePooling2D()(x)
    out = Dense(1, name='loss')(x)
    model = Model(input=input_img, output=[out])
    plot(model, to_file='SqueezeNet52.png', show_shapes=True)
    
    
    model.compile(optimizer=Adam(lr=1e-1), loss='mse')
    return model
model = squeeze_model_52()
model.summary()
Explanation: We are now ready to go into creating our model!
Main goals for us to think about while creating a model
Is it efficient for the task at hand?
If this was to be downloaded onto hardware in a car what would the power consumption and usuability be like?
There is an interesting paper called Deep Compression, which we don't implement here but it is food for thought and shows that model can be tiny and still work!
I started with a modified comma.ai model and I had a successful model, but it needed 300,000 trainable parameters. Reading the Deep compression paper and a blog post titled Self-driving car in a simulator with a tiny neural network I quickly realized that we can do better. The blog post linked shows a 63 parameter model that uses tiny images and a small network to get a stable model. I wanted to experiment and see if I could make it smaller and still stable. The solution developed over a number of experiments is a modified SqueezeNet implementation. With a squeeze net you get three addtional hyperparameters that are used to generate the fire module:
S1x1: Number of 1x1 kernals to use in the squeeze layer within the fire module
E1x1: Number of 1x1 kernals to use in the expand layer within the fire module
E3x3: Number of 3x3 kernals to use in the expand layer within the fire module
Fire Module Zoomed In
<div align="center">
  <img src="./images/fire_module.png"><br><br>
</div>
End of explanation
class CustomEarlyStop(Callback):
    
    Custom Callback that stops the epoch when val_loss reachs 0.3
    This callback assumes you are logging val loss
    
    def __init__(self, monitor='val_loss'):
        super(CustomEarlyStop, self).__init__()
        self.monitor = monitor
        
   
    def on_epoch_end(self, epoch, logs=None):
        val_loss = logs.get(self.monitor)
        if val_loss <= 0.039:
            print("\nEarly Stop on Epoch {0} with Val_loss {1}\n".format(epoch,val_loss))
            self.model.stop_training = True
Explanation: Here is the final 52 param model that we are using for this project.
<div align="left">
  </br>
  <img src="./images/SqueezeNet52.png"><br><br>
</div>
Through some experiments I know that once the loss reach around .03 we have a model that can drive in the simulator so we use a custom early termination callback to stop the training!
End of explanation
#This is a callback that works with https://github.com/fchollet/hualos and gives you simple vis of the training loss
remote = RemoteMonitor(headers=None)
early_stop = CustomEarlyStop(monitor='val_loss')
h = model.fit(x=features,y=labels,verbose=1,batch_size=128,nb_epoch=50,validation_split=0.3, callbacks=[remote, early_stop])
Explanation: To run the model training below you'll need to download and run the vis tool
Simply do the following:
git clone https://github.com/fchollet/hualos
cd into the directory
run python api.py
visit localhost:9000
run the training below and it'll produce the loss plot you see below!
Click RemoteMonitor for more information on how this callback works!
End of explanation
filename = 'model'#"squeezemodel{0}".format(squeezemodel.count_params())
model.save_weights(filename+".h5", True)
with open(filename+'.json', 'w') as outfile:
    json.dump(model.to_json(), outfile)
    
print("Saved model weights and configuration")
Explanation: Notice the loss doesn't change greatly and because we use an aggressive learning rate we could benefit from early termination.
This model was tiny (52 params!) and we are training it on a 2012 macbook air; looks like we could stop early and we could do this by writing our own keras callback to monitor and early terminate specifically for the val_loss ~0.3!
 Note: this is from a previous run without early termination 
<div align="left">
  </br>
  <img src="./images/loss_plot.png"><br><br>
</div>
End of explanation
<END_TASK>