<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 Real Life Example
Step1:  Loading and Preparing Data
Step3:  Big Kudos to Waleed Abdulla for providing the initial idea and many of the functions used to prepare and display the images
Step4:  Let's start with creating a minimal model that overfits on a very small training set
http
Step5:  This is how overfitting looks like in the Metrics
Accuracy
Validation Accuracy
Step6:  Hands-On
Step7:  How Metrics might look like when training 500 epochs with given full model
Training size makes this a little bit hard to interpret. Might look different for different random split.
Accuracy
Validation Accuracy
Step8:  What images does it work well on?
<ASSISTANT_TASK:>
Python Code:
import warnings
warnings.filterwarnings('ignore')
%matplotlib inline
%pylab inline
import matplotlib.pylab as plt
import numpy as np
from distutils.version import StrictVersion
import sklearn
print(sklearn.__version__)
assert StrictVersion(sklearn.__version__ ) >= StrictVersion('0.18.1')
import tensorflow as tf
tf.logging.set_verbosity(tf.logging.ERROR)
print(tf.__version__)
assert StrictVersion(tf.__version__) >= StrictVersion('1.1.0')
import keras
print(keras.__version__)
assert StrictVersion(keras.__version__) >= StrictVersion('2.0.6')
Explanation: Real Life Example: Classifying Speed Limit Signs
End of explanation
!ls -l speed-limit-signs
!cat speed-limit-signs/README.md
Explanation: Loading and Preparing Data
End of explanation
import os
import skimage.data
import skimage.transform
from keras.utils.np_utils import to_categorical
import numpy as np
def load_data(data_dir, type=".ppm"):
    num_categories = 6
    # Get all subdirectories of data_dir. Each represents a label.
    directories = [d for d in os.listdir(data_dir) 
                   if os.path.isdir(os.path.join(data_dir, d))]
    # Loop through the label directories and collect the data in
    # two lists, labels and images.
    labels = []
    images = []
    for d in directories:
        label_dir = os.path.join(data_dir, d)
        file_names = [os.path.join(label_dir, f) for f in os.listdir(label_dir) if f.endswith(type)]
        # For each label, load it's images and add them to the images list.
        # And add the label number (i.e. directory name) to the labels list.
        for f in file_names:
            images.append(skimage.data.imread(f))
            labels.append(int(d))
    images64 = [skimage.transform.resize(image, (64, 64)) for image in images]
    return images64, labels
# Load datasets.
ROOT_PATH = "./"
original_dir = os.path.join(ROOT_PATH, "speed-limit-signs")
images, labels = load_data(original_dir, type=".ppm")
import matplotlib
import matplotlib.pyplot as plt
def display_images_and_labels(images, labels):
    Display the first image of each label.
    unique_labels = set(labels)
    plt.figure(figsize=(15, 15))
    i = 1
    for label in unique_labels:
        # Pick the first image for each label.
        image = images[labels.index(label)]
        plt.subplot(8, 8, i)  # A grid of 8 rows x 8 columns
        plt.axis('off')
        plt.title("Label {0} ({1})".format(label, labels.count(label)))
        i += 1
        _ = plt.imshow(image)
display_images_and_labels(images, labels)
# again a little bit of feature engeneering
y = np.array(labels)
X = np.array(images)
from keras.utils.np_utils import to_categorical
num_categories = 6
y = to_categorical(y, num_categories)
Explanation: Big Kudos to Waleed Abdulla for providing the initial idea and many of the functions used to prepare and display the images: https://medium.com/@waleedka/traffic-sign-recognition-with-tensorflow-629dffc391a6#.i728o84ib
End of explanation
from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.9, random_state=42, stratify=y)
X_train.shape, y_train.shape
Explanation: Let's start with creating a minimal model that overfits on a very small training set
http://cs231n.github.io/neural-networks-3/#sanitycheck
End of explanation
# full architecture
# %load https://djcordhose.github.io/ai/fragments/vgg_style_no_dropout.py
# my sample minimized architecture
# %load https://djcordhose.github.io/ai/fragments/vgg_style_no_dropout_overfitting.py
model = Model(input=inputs, output=predictions)
model.summary()
model.compile(optimizer='adam',
              loss='categorical_crossentropy',
              metrics=['accuracy'])
# Determines how many samples are using for training in one batch
# Depends on harware GPU architecture, set as high as possible (this works well on K80)
BATCH_SIZE = 500
%time model.fit(X_train, y_train, epochs=100, validation_split=0.2, batch_size=BATCH_SIZE)
Explanation: This is how overfitting looks like in the Metrics
Accuracy
Validation Accuracy
End of explanation
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42, stratify=y)
# https://keras.io/callbacks/#tensorboard
tb_callback = keras.callbacks.TensorBoard(log_dir='./tf_log')
# To start tensorboard
# tensorboard --logdir=/mnt/c/Users/olive/Development/ml/tf_log
# open http://localhost:6006
early_stopping_callback = keras.callbacks.EarlyStopping(monitor='val_loss', patience=50, verbose=1)
checkpoint_callback = keras.callbacks.ModelCheckpoint('./model-checkpoints/weights.epoch-{epoch:02d}-val_loss-{val_loss:.2f}.hdf5');
keras.layers.Dropout?
# full architecture with dropout
# %load https://djcordhose.github.io/ai/fragments/vgg_style_dropout.py
# my sample minimized architecture
# %load https://djcordhose.github.io/ai/fragments/vgg_style_dropout_minmal.py
model = Model(input=inputs, output=predictions)
model.summary()
model.compile(optimizer='adam',
              loss='categorical_crossentropy',
              metrics=['accuracy'])
!rm -r tf_log
%time model.fit(X_train, y_train, epochs=500, batch_size=BATCH_SIZE, validation_split=0.2, callbacks=[tb_callback, early_stopping_callback])
# %time model.fit(X_train, y_train, epochs=500, batch_size=BATCH_SIZE, validation_split=0.2, callbacks=[tb_callback])
# %time model.fit(X_train, y_train, epochs=500, batch_size=BATCH_SIZE, validation_split=0.2)
train_loss, train_accuracy = model.evaluate(X_train, y_train, batch_size=BATCH_SIZE)
train_loss, train_accuracy
test_loss, test_accuracy = model.evaluate(X_test, y_test, batch_size=BATCH_SIZE)
test_loss, test_accuracy
Explanation: Hands-On: Create a minimal model
Step #1: Simplify the given architecture until you can no longer overfit on the small training set
reduce number of epochs while training to 50 or even less to have quick experinemtation cycles
reduce number of layers
reduce number of feature channels
make sure your modell actualy has less parameters than the original one (was 4,788,358)
if you need a special challenge you can write your model from scratch (you can always reload the original one using the prepared %load)
Now we see that the model at least has the basic capacity for the task, we have to get rid of the overfitting
How to avoid Overfitting using Dropouts
A Dropout Layers blacks out a certain percentage of input neurons
Which each update of weihts during training other neurons are chosen
Hope is to train different parts of the network with each iteration avoiding overfitting
Dropout rate typically between 40% and 75%
VGG adds Dropout after each convolutional block and after fc layer
x = Dropout(0.5)(x)
this only applies for training phase, in prediction there is no such layer  
Step #2: Train on the complete training set and make sure to still avoid overfitting by optimizing for val-acc
train on the complete training set
add dropout of 50% as described above
gradually make your model more complex until you have minimized overfitting
90% and more of validation accuracy are possible
again, reduce the number of epochs of your model to make it trainable on your hardware (100 might work well)
if it does not show signs of converging early on, it is likely not complex enough
you can also start with a pre-defined architecture and make this less compelx (again using the prepared %load)
save the trained model for later comparison
End of explanation
# model.save('conv-vgg.hdf5')
model.save('conv-simple.hdf5')
!ls -lh
# https://transfer.sh/
# Saved for 14 days
# !curl --upload-file conv-vgg.hdf5 https://transfer.sh
!curl --upload-file conv-simple.hdf5 https://transfer.sh
# pre-trained model
# acc: 0.98- val_acc: 0.89
# https://transfer.sh/DuZA7/conv-simple.hdf5
Explanation: How Metrics might look like when training 500 epochs with given full model
Training size makes this a little bit hard to interpret. Might look different for different random split.
Accuracy
Validation Accuracy
End of explanation
import random
# Pick 10 random images for test data set
random.seed(42) # to make this deterministic
sample_indexes = random.sample(range(len(X_test)), 10)
sample_images = [X_test[i] for i in sample_indexes]
sample_labels = [y_test[i] for i in sample_indexes]
ground_truth = np.argmax(sample_labels, axis=1)
ground_truth
X_sample = np.array(sample_images)
prediction = model.predict(X_sample)
predicted_categories = np.argmax(prediction, axis=1)
predicted_categories
# Display the predictions and the ground truth visually.
def display_prediction (images, true_labels, predicted_labels):
    fig = plt.figure(figsize=(10, 10))
    for i in range(len(true_labels)):
        truth = true_labels[i]
        prediction = predicted_labels[i]
        plt.subplot(5, 2,1+i)
        plt.axis('off')
        color='green' if truth == prediction else 'red'
        plt.text(80, 10, "Truth:        {0}\nPrediction: {1}".format(truth, prediction), 
                 fontsize=12, color=color)
        plt.imshow(images[i])
display_prediction(sample_images, ground_truth, predicted_categories)
Explanation: What images does it work well on?
End of explanation
<END_TASK>