<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 Convolutional Neural Networks
<div class="alert alert-warning">
In this notebook we will become familiar with a type of *layer* for artificial neural networks called convolutional layers. The data we will attempt to model using these types of networks will be images.
</div>
Images
For a computer an image is a matrix of data, where each pixel is represented by one or more values
Step2:  Data Pre-Processing
Before we start classifying digits we need to pre-process the data.
Your first task is to create a function that normalises 8-bit images from [0,255] to [0,1]
Step3:  Task II
Step5:  Target Pre-Processing
To classify our digits we need to use one-hot encoding to represent the target outputs. One-hot enconding is a robust yet simple solution to represent multi-categorical targets. 
This enconding is an ideal representation to train a model using gradient descent algorithm with the softmax function we discussed in a previous notebook.
Example of one-hot encoding
Here's an example of how a one-hot encoding scheme looks like
Step6:  Now that we have both added an extra dimension to the input data as well as one-hot encoded the target values, let's take a look at the shapes of the data matrices.
Step7:  Build an Artificial Neural Network with Convolutions and Max-Pooling
Convolutions
If you have the task of recognising cats in an image, you might want to recognise / classify the animal regardless of its position. To do that we rely on a statistical fact
Step8:  <div class="alert alert-info">
  <strong>In the following code snippet we will
Step9:  Task V
Step10:  Should We Use Max-Pooling?
There is a recent discussion if max-pooling is a good solution for reducing the amount of data between layers of a network. Some recent approaches show that a similar, and sometimes better performance, can be achieved by using convolutions with strides larger than 1.
Getting rid of pooling. Many people dislike the pooling operation and think that we can get away without it. For example, Striving for Simplicity
Step11:  <div class="alert alert-info">
  <strong>In the following code snippet we will
Step12:  Task VII
Step13:  CIFAR
The following explanantion of Cifar 10 comes from official cifar page
Step14:  Task VIII
Step15:  Task IX
Step16:  Task X
Step17:  Task XI
<ASSISTANT_TASK:>
Python Code:
# Plots will be displaying plots within the notebook
%matplotlib notebook
import matplotlib.pyplot as plt
from matplotlib.ticker import MaxNLocator
# NumPy is a package for manipulating N-dimensional array objects 
import numpy as np
# Pandas is a data analysis package
import pandas as pd
#Library To test/verify some tasks
import problem_unittests as tests # Used to test ouw anwsers
# Mnist wrapper
from keras.datasets import mnist
# Code to load the data
(x_train, y_train), (x_test, y_test) = mnist.load_data()
# Print data shape
print('Shape of x_train {}'.format(x_train.shape))
print('Shape of y_train {}'.format(y_train.shape))
print('Shape of x_test {}'.format(x_train.shape))
print('Shape of y_test {}'.format(y_train.shape))
# Code to plot the 5th training sample.
fig,ax1 = plt.subplots(1,1, figsize=(7, 7))
ax1.imshow(x_train[5], cmap='gray')
title = 'Target = {}'.format(y_train[5])
ax1.set_title(title)
ax1.grid(which='Major')
ax1.xaxis.set_major_locator(MaxNLocator(28))
ax1.yaxis.set_major_locator(MaxNLocator(28))
fig.canvas.draw()
time.sleep(0.1)
Explanation: Convolutional Neural Networks
<div class="alert alert-warning">
In this notebook we will become familiar with a type of *layer* for artificial neural networks called convolutional layers. The data we will attempt to model using these types of networks will be images.
</div>
Images
For a computer an image is a matrix of data, where each pixel is represented by one or more values:
Matrix with one value per pixel  =  greyscale images
<img src="./resources/lincoln_pixel_values.png" alt="Grayscale Image" width="700">image_source
Matrix with three values per pixel  =  color images
<img src="./resources/color_images.png" alt="Decomposition of a color image" width="400">
image_source
The MNIST Dataset
We had a brief look at this dataset in the previous notebook, and here we will go through it again with much more detail. As before, the MNIST database (Modified National Institute of Standards and Technology database) is a multiclass classification problem where we are tasked with classifying a digit ($0-9$) based on a $28\times 28$ greyscale image:
The MNIST database of handwritten digits, available from this page, has a training set of 60,000 examples, and a test set of 10,000 examples. It is a subset of a larger set available from NIST. The digits have been size-normalized and centered in a fixed-size image.
It is a good database for people who want to try learning techniques and pattern recognition methods on real-world data while spending minimal efforts on preprocessing and formatting.
source
In the following example we will load data from MNIST.
input $\rightarrow$ 70000 samples of vectors
Each vector has 784 dimensions
Here presented as $28\times 28$ matrices $\rightarrow$ Greyscale images
target $\rightarrow$ 70000 integers indicating a digit from 0 to 9
<div class="alert alert-info">
<strong>In the following snippet of code we will:</strong>
<ul>
  <li>Load the MNIST dataset</li>
  <li>Plot the 5th sample of the training set</li>
</ul>
</div>
End of explanation
def normalise_images(images):
    Normalise input images.
    
    # Normalise image here
    return images
### Do *not* modify the following lines ###
tests.test_normalize_images(normalise_images)
# Normalize the data for future use
x_train = normalise_images(x_train)
x_test = normalise_images(x_test)
Explanation: Data Pre-Processing
Before we start classifying digits we need to pre-process the data.
Your first task is to create a function that normalises 8-bit images from [0,255] to [0,1]:
Task I:  Implement an Image Normalisation Function
<div class="alert alert-success">
**Task**: Implement a function that normalises the images to the interval [0,1].
<ul>
  <li>Inputs are integers in the interval [0,255]</li>
  <li>Outputs should be floats in the interval [0,1]</li>
</ul>
</div>
End of explanation
# Write your code here
x_train = None
x_test = None
### Do *not* modify the following lines ###
print('Shape of x_train {}'.format(x_train.shape))
print('Shape of y_train {}'.format(y_train.shape))
print('Shape of x_test {}'.format(x_test.shape))
print('Shape of y_test {}'.format(y_test.shape))
Explanation: Task II:  Expand the Dimension of the Input
When we loaded the MNIST dataset each digit was represented by a matrix of size $(28, 28)$. However, the artificial neural network we will be building uses the concept of colour channels and feature maps even for greyscale images. This means that we have to transform $(28, 28)$ to $(28, 28, 1)$.
<div class="alert alert-success">
**Task**: Write a piece of code that add a new dimestion to `x_train` and `x_test`.
<ul>
  <li>The shape of `x_train` should be $(60000, 28, 28, 1)$</li>
  <li>The shape of `x_test` should be $(10000, 28, 28, 1)$</li>
</ul>
Take a look at [numpy.expand_dims()](https://docs.scipy.org/doc/numpy/reference/generated/numpy.expand_dims.html) for how you might do this.
</div>
End of explanation
def one_hot(vector, number_classes):
    Return a one-hot encoded matrix given the argument vector.
    
    # Where we will store our one-hots
    one_hot = []
    # One-hot encode `vector` here
    # Transform list to numpy array and return it
    return np.array(one_hot)
### Do *not* modify the following line ###
tests.test_one_hot(one_hot)
# One-hot encode the MNIST target values
y_train = one_hot (y_train, 10)
y_test = one_hot(y_test, 10)
Explanation: Target Pre-Processing
To classify our digits we need to use one-hot encoding to represent the target outputs. One-hot enconding is a robust yet simple solution to represent multi-categorical targets. 
This enconding is an ideal representation to train a model using gradient descent algorithm with the softmax function we discussed in a previous notebook.
Example of one-hot encoding
Here's an example of how a one-hot encoding scheme looks like:
<img src="./resources/one_hot.png" alt="One-Hot encoding" width="650">
The core idea is that you transform multi-categorical data to a combination of several single class(es). By doing this we can, for each example, see whether it belongs to any class, where 1 indicates that it does and 0 otherwise.
Task III:  Implement a Function for One-Hot Encoding
<div class="alert alert-success">
**Task**: Implement a function that one-hot encodes a vector of numbers to a matrix of $K$ classes:
<ul>
  <li>The first argument is vector with $N$ samples (dimensions)</li>
  <li>The second argument is a number $K$ signifying the number of classes</li>
  <li>For each sample of the vector you will create an array with $K$ dimensions</li>
  <li>The one-hot encoded matrix should have zeros on all positions expect on the position indicated by the current sample in the input vector</li>
</ul>
</div>
Try to implement this function by yourself. If you have doubts ask for help
If you are running out of time use keras.utils.to_categorical(vector, number_classes) like we did in the previous notebook and come back to this task later
End of explanation
print('Shape of x_train {}'.format(x_train.shape))
print('Shape of y_train {}'.format(y_train.shape))
print('Shape of x_test {}'.format(x_train.shape))
print('Shape of y_test {}'.format(y_train.shape))
Explanation: Now that we have both added an extra dimension to the input data as well as one-hot encoded the target values, let's take a look at the shapes of the data matrices.
End of explanation
# Import Keras library
import keras
from keras.models import Model
from keras.layers import *
def net_1(sample_shape, nb_classes):
    # Define the network input to have `sample_shape´ shape
    input_x = None
    
    # Create network internals here
    x = None
    
    # Dense `nb_classes`
    probabilities = Dense(nb_classes, activation='softmax')(x)
    
    # Define the output
    model = Model(inputs=input_x, outputs=probabilities)
    return model
Explanation: Build an Artificial Neural Network with Convolutions and Max-Pooling
Convolutions
If you have the task of recognising cats in an image, you might want to recognise / classify the animal regardless of its position. To do that we rely on a statistical fact: natural images are stationary source.
So, if we calculate a statistic for some location in the input image, then that statistic might also be valuable to calculate at some other location. One can exploit this property to define small networks that learn features that can be applied on different parts of an image.
Convolutional neural networks employs these aspects to create very efficient neural networks.
In case you want to watch a short video (has captions) walking through the concepts behind convolutional networks, take a look at the following YouTube link:
Udacity - Convolutional Networks
Sliding windows
A sliding window defines a small region of interest in an image.
The region of interested is used to scan the whole image as shown in the following animation:
<img src="./resources/sliding_window_example.gif" alt="Sliding window on a greyscale image" width="200">
If we use the sliding window to define what is the input seen by a small neural network, we have a so called convolution.
Assuming we have a color image, and a small neural network with $k$ outputs: for every possible position of the sliding window we will have $k$ outputs.
<img src="./resources/conv.png" alt="Output of a convolution at for a given sliding window placement" width="300">
After the sliding window has scanned the whole image you have 3 dimensional matrix that can be investigated further.
<img src="./resources/conv2.png" alt="Output of a convolution at for a given sliding window placement" width="300">
Convolution Example
Let's assume that we have an image of $5 \times 5=25$ pixels:
$$
\begin{equation}
\begin{array}{|c|c|c|c|c|}
  \hline
  1 & 1 & 1 & 0 & 0 \ 
  \hline
  0 & 1 & 1 & 1 & 0\
  \hline
  0 & 0 & 1 & 1 & 1\
  \hline
  0 & 0 & 1 & 1 & 0\
  \hline
  0 & 1 & 1 & 0 & 0\
  \hline
\end{array}
\end{equation}
$$
Assume that we define a small neural network that has  $3 \times 3$ weights and a single output.
The weight matrix is
$$
\begin{equation}
\begin{array}{|c|c|c|}
  \hline
  1 & 0 & 1 \ 
  \hline
  0 & 1 & 0\
  \hline
  1 & 0 & 1\
  \hline
\end{array}
\end{equation}
$$
By feed-forwarding the network using a $3 \times 3$  sliding window we get the following  convolved features (also know as feature map):
$$
\begin{equation}
\begin{array}{|c|c|c|}
  \hline
  4 & 3 & 4 \ 
  \hline
  2 & 4 & 3\
  \hline
  2 & 3 & 4\
  \hline
\end{array}
\end{equation}
$$
<img src="./resources/Convolution_schematic.gif" alt="Sliding window" width="500">
source
The number of so-called feature maps produced will depend on the number of outputs of the neural network. In this case we have just one feature map.
Stride and Padding
We can use padding and strides to achive different shapes on the feature maps.
Below are four animations that showcases the convolution operation on an input matrix using different paddings and strides:
<table style="width:100%">
  <tr>
    <td><img src="./resources/no_padding_no_strides.gif"></td>
    <td><img src="./resources/no_padding_strides.gif"></td>
    <td><img src="./resources/same_padding_no_strides.gif"></td>
    <td><img src="./resources/padding_strides.gif"></td>
  </tr>
  <tr>
    <td>Valid padding, stride=1</td>
    <td>Valid padding, stride=2</td>
    <td>Same Padding, stride =1</td>
    <td>Padding =1 , stride =2</td>
  </tr>
</table>
source
Paper
Computing the Size of the Convolutions
To compute the size of the feature map resulting from a convolution we need to know the input size, the size of the kernel (filter), the stride, and the padding:
$$
\begin{equation}
output = \frac{1}{stride} (input -  kernel + 2 * padding) + 1
\end{equation}
$$
The height can be calculated like this:
$$
\begin{equation}
height_{new} =  \frac{1}{stride} (height_{input} - height_{kernel} + 2 * padding)+1
\end{equation}
$$
The width can be calculated like this:
$$
\begin{equation}
widht_{new} =  \frac{1}{stride} (widht_{input} - widht_{kernel} + 2 * padding)+1
\end{equation}
$$
Example :
Let us assume that we have an image that is $5 \times 5$. If we pad this image with a single pixel and then convolve it with a $3 \times 3$ kernel using a stride of $2$, we get the following feature map:
<img src="./resources/odd.gif" alt="Examples math" width="300">
We can compute the output size using the equations above:
$$
\begin{equation}
\begin{aligned}
height_{new} &= \frac{1}{stride} (height_{input} - height_{kernel} + 2 * padding) + 1 \
&= \frac{1}{2} (5 - 3 + 2 * 1) + 1 \
&= \frac{1}{2}(4)+1 \
&= 3
\end{aligned}
\end{equation}
$$
Pooling
It has become common practice to use pooling layer between convolutional layers.
Sucessful convolutional neural networks like alexnet, VGG16, VGG19 employed this technique.
Pooling layers also depend on sliding windows, but instead of using the window as inputs for neurons, the sliding input data goes through a max, mean, or some other operator.
A Max-Pooling Example
<img src="./resources/pooling.gif" alt="Max Pooling" width="300">
Max-pooling has several advantages:
* If you have images where the same class has similar images with small shifts in the pixel position, max pooling will mitigate small translations
* It introduces zero parameters to the model since the max and mean operators are fixed functions that do not depend on weights
* It reduces the amount of data that need to be processed in the next layer, while assuring some pixel translation invariance.
* They are normally used with zero padding (aka valid padding).
* They follow the same dimensional maths as convolutions
You can read more about pooling operators here
Implementing a Convolutional Network
We will implement the following convolution network:
<img src="./resources/mnist_net.png" alt="CNN" width="1280">
The components of this network can be seen below:
Define an input:
input_x = Input(shape=sample_shape)
sample_shape is an input parameter
Generate 32 kernel maps using a convolutional layer:
The convolution uses a $3 \times 3$ kernel, stride 1, valid padding, and [ReLU](https://en.wikipedia.org/wiki/Rectifier_(neural_networks) activation
Use Conv2D from Keras
output_layer = Conv2D(PARAMETERS)(input_layer)
Generate 64 kernel maps using a convolutional layer:
The convolution uses a $3 \times 3$ kernel, stride 1, valid padding, and [ReLU](https://en.wikipedia.org/wiki/Rectifier_(neural_networks) activation
Use Conv2D
output_layer = Conv2D(PARAMETERS)(input_layer)
Reduce the feature maps using max-pooling:
The max-pooling should us a $2 \times 2$ kernel, stride 1, and valid padding
Use MaxPooling2D
output_layer = MaxPooling2D(PARAMETERS)(input_layer)
Flatten the feature map:
Flatten
Fully-connected, i.e. Dense, to 128 dimensions:
[ReLU](https://en.wikipedia.org/wiki/Rectifier_(neural_networks) activation
Dense
Fully-connected, i.e. Dense, to $K$ classes (argument) dimensions:
Softmax activation
Dense
Task IV:  Implement a Convolutional Neural Network Model
It is time to implement our first convolutional neural network.
<div class="alert alert-success">
<strong>Task:</strong> Create a function `net_1()` that implements the network specified above.
Make sure to refer back to earlier notebooks if you are unsure about what to do.
</div>
End of explanation
# Shape of sample
sample_shape = x_train[0].shape 
# Construct net
model = net_1(sample_shape, 10)
model.summary()
Explanation: <div class="alert alert-info">
  <strong>In the following code snippet we will:</strong>
<ul>
  <li>Create the network using the function you just made</li>
  <li>Display a summary of the network</li>
</ul>
</div>
End of explanation
# Define hyperparameters
batch_size = None
epochs = None
### Do *not* modify the following lines ###
# There is no learning rate because we are using the recommended
# values for the Adadelta optimiser more information here:
# https://keras.io/optimizers/
# We need to compile our model
model.compile(loss='categorical_crossentropy',
              optimizer='Adadelta',
              metrics=['accuracy'])
# Train
logs = model.fit(x_train, y_train,
                 batch_size=batch_size,
                 epochs=epochs,
                 verbose=2,
                 validation_split=0.1)
# Plot our losses and accuracy
fig, ax = plt.subplots(1,1)
pd.DataFrame(logs.history).plot(ax=ax)
ax.grid(linestyle='dotted')
ax.legend()
plt.show()
# Assess performance
print('='*80)    
print('Assesing Test dataset...')
print('='*80)    
score = model.evaluate(x_test, y_test, verbose=0)
print('Test loss:', score[0])
print('Test accuracy:', score[1])
Explanation: Task V: Define Hyperparameters and Train the Network
We need to define hyperparameters so our network can learn.
<div class="alert alert-success">
<strong>Task:</strong> Tune in the hyper-parameters until your `loss` and `val_loss` are both converging to low numbers:
<ul>
  <li>Batch size</li>
  <li>Number of training epochs</li>
</ul>
</div>
Keep in mind that training these kinds of networks will take longer than the ones we have looked at so far.
End of explanation
def net_2(sample_shape, nb_classes):
    # Define the network input to have `sample_shape` shape
    input_x = None
    
    # Create network internals here
    x = None
    # Dense number_classes
    probabilities = Dense(nb_classes, activation='softmax')(x)
    # Define the output
    model = Model(inputs=input_x, outputs=probabilities)
    return model
Explanation: Should We Use Max-Pooling?
There is a recent discussion if max-pooling is a good solution for reducing the amount of data between layers of a network. Some recent approaches show that a similar, and sometimes better performance, can be achieved by using convolutions with strides larger than 1.
Getting rid of pooling. Many people dislike the pooling operation and think that we can get away without it. For example, Striving for Simplicity: The All Convolutional Net proposes to discard the pooling layer in favor of architecture that only consists of repeated CONV layers. To reduce the size of the representation they suggest using larger stride in CONV layer once in a while. Discarding pooling layers has also been found to be important in training good generative models, such as variational autoencoders (VAEs) or generative adversarial networks (GANs). It seems likely that future architectures will feature very few to no pooling layers.
source
Task VI: Implement a Convolutional Network Without Max-Pooling
Implement a convolutional neural network without pooling layers:
<img src="./resources/mnist_net2.png" alt="CNN" width="1280">
<div class="alert alert-success">
<strong>Task:</strong> Replicate the network we made before (`net_1()`), but this time:
<ul>
  <li>Remove max pooling and add stride(s) of 2 to the second convolution block (see [Conv2D](https://keras.io/layers/convolutional/))</li>
</ul>
</div>
End of explanation
# Shape of sample
sample_shape = x_train[0].shape 
# Construct net
model = net_2(sample_shape, 10)
model.summary()
Explanation: <div class="alert alert-info">
  <strong>In the following code snippet we will:</strong>
<ul>
  <li>Create the network using the function you just made</li>
  <li>Display a summary of the network</li>
</ul>
</div>
End of explanation
# Define hyperparameters
batch_size = None
epochs = None
### Do *not* modify the following lines ###
# As always we need to compile our model
model.compile(loss='categorical_crossentropy',
              optimizer='Adadelta',
              metrics=['accuracy'])
# Train
LOGS = model.fit(x_train, y_train,
          batch_size=batch_size,
          epochs=epochs,
          verbose=2,
          validation_split = 0.1,)
# Plot our losses and accuracy
fig, ax = plt.subplots(1,1)
pd.DataFrame(logs.history).plot(ax=ax)
ax.grid(linestyle='dotted')
ax.legend()
fig.canvas.draw()
# Assess performance
print('='*80)    
print('Assesing Test dataset...')
print('='*80)    
score = model.evaluate(x_test, y_test, verbose=0)
print('Test loss:', score[0])
print('Test accuracy:', score[1])
Explanation: Task VII: Define Hyperparameters and Train the New Network
As before, we need to define some hyperparameters and train the network. Feel free to reuse the hyperparameters you found before.
<div class="alert alert-success">
<strong>Task:</strong> Tune in the hyper-parameters until your `loss` and `val_loss` are both converging to low numbers:
<ul>
  <li>Batch size</li>
  <li>Number of training epochs</li>
</ul>
</div>
End of explanation
from keras.datasets import cifar10
# The data, shuffled and split between train and test sets:
(x_train, y_train), (x_test, y_test) = cifar10.load_data()
print('x_train shape:', x_train.shape)
print(x_train.shape[0], 'train samples')
print(x_test.shape[0], 'test samples')
target_2_class = {0:'airplane',
                  1:'automobile',
                  2:'bird',
                  3:'cat',
                  4:'deer',
                  5:'dog',
                  6:'frog',
                  7:'horse',
                  8:'ship',
                  9:'truck'}
# Code to plot the 5th training sample.
fig,ax1 = plt.subplots(1,1, figsize=(7,7))
ax1.imshow(x_train[5])
target = y_train[5][0]
title = 'Target is {} - Class {}'.format(target_2_class[target],target )
ax1.set_title(title)
ax1.grid(which='Major')
ax1.xaxis.set_major_locator(MaxNLocator(32))
ax1.yaxis.set_major_locator(MaxNLocator(32))
fig.canvas.draw()
time.sleep(0.1)
print('Shape of x_train {}'.format(x_train.shape))
print('Shape of y_train {}'.format(y_train.shape))
print('Shape of x_test {}'.format(x_train.shape))
print('Shape of y_test {}'.format(y_train.shape))
Explanation: CIFAR
The following explanantion of Cifar 10 comes from official cifar page:
The CIFAR-10 and CIFAR-100 are labeled subsets of the <a href="http://people.csail.mit.edu/torralba/tinyimages/">80 million tiny images</a> dataset. They were collected by Alex Krizhevsky, Vinod Nair, and Geoffrey Hinton.
The CIFAR10 Dataset
The CIFAR-10 dataset consists of 60000 $32 \times 32$ colour images in 10 classes, with 6000 images per class. There are 50000 training images and 10000 test images.
The dataset is divided into five training batches and one test batch, each with 10000 images. The test batch contains exactly 1000 randomly-selected images from each class. The training batches contain the remaining images in random order, but some training batches may contain more images from one class than another. Between them, the training batches contain exactly 5000 images from each class.</li>
Here are the classes in the dataset, as well as 10 random images from each:
<table>
    <tr>
        <td class="cifar-class-name">airplane</td>
        <td><img src="https://www.cs.toronto.edu/~kriz/cifar-10-sample/airplane1.png" class="cifar-sample" /></td>
        <td><img src="https://www.cs.toronto.edu/~kriz/cifar-10-sample/airplane2.png" class="cifar-sample" /></td>
        <td><img src="https://www.cs.toronto.edu/~kriz/cifar-10-sample/airplane3.png" class="cifar-sample" /></td>
        <td><img src="https://www.cs.toronto.edu/~kriz/cifar-10-sample/airplane4.png" class="cifar-sample" /></td>
        <td><img src="https://www.cs.toronto.edu/~kriz/cifar-10-sample/airplane5.png" class="cifar-sample" /></td>
        <td><img src="https://www.cs.toronto.edu/~kriz/cifar-10-sample/airplane6.png" class="cifar-sample" /></td>
        <td><img src="https://www.cs.toronto.edu/~kriz/cifar-10-sample/airplane7.png" class="cifar-sample" /></td>
        <td><img src="https://www.cs.toronto.edu/~kriz/cifar-10-sample/airplane8.png" class="cifar-sample" /></td>
        <td><img src="https://www.cs.toronto.edu/~kriz/cifar-10-sample/airplane9.png" class="cifar-sample" /></td>
        <td><img src="https://www.cs.toronto.edu/~kriz/cifar-10-sample/airplane10.png" class="cifar-sample" /></td>
    </tr>
    <tr>
        <td class="cifar-class-name">automobile</td>
        <td><img src="https://www.cs.toronto.edu/~kriz/cifar-10-sample/automobile1.png" class="cifar-sample" /></td>
        <td><img src="https://www.cs.toronto.edu/~kriz/cifar-10-sample/automobile2.png" class="cifar-sample" /></td>
        <td><img src="https://www.cs.toronto.edu/~kriz/cifar-10-sample/automobile3.png" class="cifar-sample" /></td>
        <td><img src="https://www.cs.toronto.edu/~kriz/cifar-10-sample/automobile4.png" class="cifar-sample" /></td>
        <td><img src="https://www.cs.toronto.edu/~kriz/cifar-10-sample/automobile5.png" class="cifar-sample" /></td>
        <td><img src="https://www.cs.toronto.edu/~kriz/cifar-10-sample/automobile6.png" class="cifar-sample" /></td>
        <td><img src="https://www.cs.toronto.edu/~kriz/cifar-10-sample/automobile7.png" class="cifar-sample" /></td>
        <td><img src="https://www.cs.toronto.edu/~kriz/cifar-10-sample/automobile8.png" class="cifar-sample" /></td>
        <td><img src="https://www.cs.toronto.edu/~kriz/cifar-10-sample/automobile9.png" class="cifar-sample" /></td>
        <td><img src="https://www.cs.toronto.edu/~kriz/cifar-10-sample/automobile10.png" class="cifar-sample" /></td>
    </tr>
    <tr>
        <td class="cifar-class-name">bird</td>
        <td><img src="https://www.cs.toronto.edu/~kriz/cifar-10-sample/bird1.png" class="cifar-sample" /></td>
        <td><img src="https://www.cs.toronto.edu/~kriz/cifar-10-sample/bird2.png" class="cifar-sample" /></td>
        <td><img src="https://www.cs.toronto.edu/~kriz/cifar-10-sample/bird3.png" class="cifar-sample" /></td>
        <td><img src="https://www.cs.toronto.edu/~kriz/cifar-10-sample/bird4.png" class="cifar-sample" /></td>
        <td><img src="https://www.cs.toronto.edu/~kriz/cifar-10-sample/bird5.png" class="cifar-sample" /></td>
        <td><img src="https://www.cs.toronto.edu/~kriz/cifar-10-sample/bird6.png" class="cifar-sample" /></td>
        <td><img src="https://www.cs.toronto.edu/~kriz/cifar-10-sample/bird7.png" class="cifar-sample" /></td>
        <td><img src="https://www.cs.toronto.edu/~kriz/cifar-10-sample/bird8.png" class="cifar-sample" /></td>
        <td><img src="https://www.cs.toronto.edu/~kriz/cifar-10-sample/bird9.png" class="cifar-sample" /></td>
        <td><img src="https://www.cs.toronto.edu/~kriz/cifar-10-sample/bird10.png" class="cifar-sample" /></td>
    </tr>
    <tr>
        <td class="cifar-class-name">cat</td>
        <td><img src="https://www.cs.toronto.edu/~kriz/cifar-10-sample/cat1.png" class="cifar-sample" /></td>
        <td><img src="https://www.cs.toronto.edu/~kriz/cifar-10-sample/cat2.png" class="cifar-sample" /></td>
        <td><img src="https://www.cs.toronto.edu/~kriz/cifar-10-sample/cat3.png" class="cifar-sample" /></td>
        <td><img src="https://www.cs.toronto.edu/~kriz/cifar-10-sample/cat4.png" class="cifar-sample" /></td>
        <td><img src="https://www.cs.toronto.edu/~kriz/cifar-10-sample/cat5.png" class="cifar-sample" /></td>
        <td><img src="https://www.cs.toronto.edu/~kriz/cifar-10-sample/cat6.png" class="cifar-sample" /></td>
        <td><img src="https://www.cs.toronto.edu/~kriz/cifar-10-sample/cat7.png" class="cifar-sample" /></td>
        <td><img src="https://www.cs.toronto.edu/~kriz/cifar-10-sample/cat8.png" class="cifar-sample" /></td>
        <td><img src="https://www.cs.toronto.edu/~kriz/cifar-10-sample/cat9.png" class="cifar-sample" /></td>
        <td><img src="https://www.cs.toronto.edu/~kriz/cifar-10-sample/cat10.png" class="cifar-sample" /></td>
    </tr>
    <tr>
        <td class="cifar-class-name">deer</td>
        <td><img src="https://www.cs.toronto.edu/~kriz/cifar-10-sample/deer1.png" class="cifar-sample" /></td>
        <td><img src="https://www.cs.toronto.edu/~kriz/cifar-10-sample/deer2.png" class="cifar-sample" /></td>
        <td><img src="https://www.cs.toronto.edu/~kriz/cifar-10-sample/deer3.png" class="cifar-sample" /></td>
        <td><img src="https://www.cs.toronto.edu/~kriz/cifar-10-sample/deer4.png" class="cifar-sample" /></td>
        <td><img src="https://www.cs.toronto.edu/~kriz/cifar-10-sample/deer5.png" class="cifar-sample" /></td>
        <td><img src="https://www.cs.toronto.edu/~kriz/cifar-10-sample/deer6.png" class="cifar-sample" /></td>
        <td><img src="https://www.cs.toronto.edu/~kriz/cifar-10-sample/deer7.png" class="cifar-sample" /></td>
        <td><img src="https://www.cs.toronto.edu/~kriz/cifar-10-sample/deer8.png" class="cifar-sample" /></td>
        <td><img src="https://www.cs.toronto.edu/~kriz/cifar-10-sample/deer9.png" class="cifar-sample" /></td>
        <td><img src="https://www.cs.toronto.edu/~kriz/cifar-10-sample/deer10.png" class="cifar-sample" /></td>
    </tr>
    <tr>
        <td class="cifar-class-name">dog</td>
        <td><img src="https://www.cs.toronto.edu/~kriz/cifar-10-sample/dog1.png" class="cifar-sample" /></td>
        <td><img src="https://www.cs.toronto.edu/~kriz/cifar-10-sample/dog2.png" class="cifar-sample" /></td>
        <td><img src="https://www.cs.toronto.edu/~kriz/cifar-10-sample/dog3.png" class="cifar-sample" /></td>
        <td><img src="https://www.cs.toronto.edu/~kriz/cifar-10-sample/dog4.png" class="cifar-sample" /></td>
        <td><img src="https://www.cs.toronto.edu/~kriz/cifar-10-sample/dog5.png" class="cifar-sample" /></td>
        <td><img src="https://www.cs.toronto.edu/~kriz/cifar-10-sample/dog6.png" class="cifar-sample" /></td>
        <td><img src="https://www.cs.toronto.edu/~kriz/cifar-10-sample/dog7.png" class="cifar-sample" /></td>
        <td><img src="https://www.cs.toronto.edu/~kriz/cifar-10-sample/dog8.png" class="cifar-sample" /></td>
        <td><img src="https://www.cs.toronto.edu/~kriz/cifar-10-sample/dog9.png" class="cifar-sample" /></td>
        <td><img src="https://www.cs.toronto.edu/~kriz/cifar-10-sample/dog10.png" class="cifar-sample" /></td>
    </tr>
    <tr>
        <td class="cifar-class-name">frog</td>
        <td><img src="https://www.cs.toronto.edu/~kriz/cifar-10-sample/frog1.png" class="cifar-sample" /></td>
        <td><img src="https://www.cs.toronto.edu/~kriz/cifar-10-sample/frog2.png" class="cifar-sample" /></td>
        <td><img src="https://www.cs.toronto.edu/~kriz/cifar-10-sample/frog3.png" class="cifar-sample" /></td>
        <td><img src="https://www.cs.toronto.edu/~kriz/cifar-10-sample/frog4.png" class="cifar-sample" /></td>
        <td><img src="https://www.cs.toronto.edu/~kriz/cifar-10-sample/frog5.png" class="cifar-sample" /></td>
        <td><img src="https://www.cs.toronto.edu/~kriz/cifar-10-sample/frog6.png" class="cifar-sample" /></td>
        <td><img src="https://www.cs.toronto.edu/~kriz/cifar-10-sample/frog7.png" class="cifar-sample" /></td>
        <td><img src="https://www.cs.toronto.edu/~kriz/cifar-10-sample/frog8.png" class="cifar-sample" /></td>
        <td><img src="https://www.cs.toronto.edu/~kriz/cifar-10-sample/frog9.png" class="cifar-sample" /></td>
        <td><img src="https://www.cs.toronto.edu/~kriz/cifar-10-sample/frog10.png" class="cifar-sample" /></td>
    </tr>
    <tr>
        <td class="cifar-class-name">horse</td>
        <td><img src="https://www.cs.toronto.edu/~kriz/cifar-10-sample/horse1.png" class="cifar-sample" /></td>
        <td><img src="https://www.cs.toronto.edu/~kriz/cifar-10-sample/horse2.png" class="cifar-sample" /></td>
        <td><img src="https://www.cs.toronto.edu/~kriz/cifar-10-sample/horse3.png" class="cifar-sample" /></td>
        <td><img src="https://www.cs.toronto.edu/~kriz/cifar-10-sample/horse4.png" class="cifar-sample" /></td>
        <td><img src="https://www.cs.toronto.edu/~kriz/cifar-10-sample/horse5.png" class="cifar-sample" /></td>
        <td><img src="https://www.cs.toronto.edu/~kriz/cifar-10-sample/horse6.png" class="cifar-sample" /></td>
        <td><img src="https://www.cs.toronto.edu/~kriz/cifar-10-sample/horse7.png" class="cifar-sample" /></td>
        <td><img src="https://www.cs.toronto.edu/~kriz/cifar-10-sample/horse8.png" class="cifar-sample" /></td>
        <td><img src="https://www.cs.toronto.edu/~kriz/cifar-10-sample/horse9.png" class="cifar-sample" /></td>
        <td><img src="https://www.cs.toronto.edu/~kriz/cifar-10-sample/horse10.png" class="cifar-sample" /></td>
    </tr>
    <tr>
        <td class="cifar-class-name">ship</td>
        <td><img src="https://www.cs.toronto.edu/~kriz/cifar-10-sample/ship1.png" class="cifar-sample" /></td>
        <td><img src="https://www.cs.toronto.edu/~kriz/cifar-10-sample/ship2.png" class="cifar-sample" /></td>
        <td><img src="https://www.cs.toronto.edu/~kriz/cifar-10-sample/ship3.png" class="cifar-sample" /></td>
        <td><img src="https://www.cs.toronto.edu/~kriz/cifar-10-sample/ship4.png" class="cifar-sample" /></td>
        <td><img src="https://www.cs.toronto.edu/~kriz/cifar-10-sample/ship5.png" class="cifar-sample" /></td>
        <td><img src="https://www.cs.toronto.edu/~kriz/cifar-10-sample/ship6.png" class="cifar-sample" /></td>
        <td><img src="https://www.cs.toronto.edu/~kriz/cifar-10-sample/ship7.png" class="cifar-sample" /></td>
        <td><img src="https://www.cs.toronto.edu/~kriz/cifar-10-sample/ship8.png" class="cifar-sample" /></td>
        <td><img src="https://www.cs.toronto.edu/~kriz/cifar-10-sample/ship9.png" class="cifar-sample" /></td>
        <td><img src="https://www.cs.toronto.edu/~kriz/cifar-10-sample/ship10.png" class="cifar-sample" /></td>
    </tr>
    <tr>
        <td class="cifar-class-name">truck</td>
        <td><img src="https://www.cs.toronto.edu/~kriz/cifar-10-sample/truck1.png" class="cifar-sample" /></td>
        <td><img src="https://www.cs.toronto.edu/~kriz/cifar-10-sample/truck2.png" class="cifar-sample" /></td>
        <td><img src="https://www.cs.toronto.edu/~kriz/cifar-10-sample/truck3.png" class="cifar-sample" /></td>
        <td><img src="https://www.cs.toronto.edu/~kriz/cifar-10-sample/truck4.png" class="cifar-sample" /></td>
        <td><img src="https://www.cs.toronto.edu/~kriz/cifar-10-sample/truck5.png" class="cifar-sample" /></td>
        <td><img src="https://www.cs.toronto.edu/~kriz/cifar-10-sample/truck6.png" class="cifar-sample" /></td>
        <td><img src="https://www.cs.toronto.edu/~kriz/cifar-10-sample/truck7.png" class="cifar-sample" /></td>
        <td><img src="https://www.cs.toronto.edu/~kriz/cifar-10-sample/truck8.png" class="cifar-sample" /></td>
        <td><img src="https://www.cs.toronto.edu/~kriz/cifar-10-sample/truck9.png" class="cifar-sample" /></td>
        <td><img src="https://www.cs.toronto.edu/~kriz/cifar-10-sample/truck10.png" class="cifar-sample" /></td>
    </tr>
</table>
<br/>
The classes are completely mutually exclusive. There is no overlap between automobiles and trucks. "Automobile" includes sedans, SUVs, things of that sort. "Truck" includes only big trucks. Neither includes pickup trucks.
In the following example we will load data from CIFAR10.
input $\rightarrow$ 60000 samples of 3072 dimensional vectors.
Here presented as $32 \times 32 \times 3$ matrices $\rightarrow$ Colour images
target $\rightarrow$ 60000 scalars indicating a class from 0 to 9
<div class="alert alert-info">
  <strong>In the following code we will:</strong>
<ul>
  <li>Load the CIFAR10 dataset</li>
  <li>Plot the 5th sample of training set</li>
</ul>
</div>
End of explanation
y_train = None
y_test = None
### Do *not* modify the following line ###
# Print data sizes
print('Shape of x_train {}'.format(x_train.shape))
print('Shape of y_train {}'.format(y_train.shape))
print('Shape of x_test {}'.format(x_train.shape))
print('Shape of y_test {}'.format(y_train.shape))
Explanation: Task VIII: One-Hot Encode the Target Values
<div class="alert alert-success">
<strong>Task:</strong> Use the `one_hot()` function you created earlier to encode:
<ul>
  <li>`y_test`</li>
  <li>`y_train`</li>
</ul>
</div>
End of explanation
x_train = None
x_test = None
Explanation: Task IX: Normalise the Images
<div class="alert alert-success">
<strong>Task:</strong> Use the `normalise_images()` function you created earlier to normalise the images in:
<ul>
  <li>`x_test`</li>
  <li>`x_train`</li>
</ul>
</div>
End of explanation
# Shape of samples
sample_shape = x_train[0].shape 
# Construct net
model = None
model.summary()
# We need to compile our model network:
model.compile(loss='categorical_crossentropy',
              optimizer='Adam',
              metrics=['accuracy'])
Explanation: Task X: Create Your Convolutional Neural Network for CIFAR10
<div class="alert alert-success">
  <strong>Task:</strong> Create a neural network model to train on CIFAR using what we have learned so far.
<ul>
  <li>Create a new network using either `net_1()` or `net_2()`</li>
  <li>Display a summary of the network</li>
  <li>Compile the model using eiter `Adadelta`, `Adagrad`, or `Adam` as your optimiser</li>
</ul>
Some of the code is filled in already so alter what you need.
</div>
End of explanation
# Build the code within this cell
Explanation: Task XI: Train Your Model
<div class="alert alert-success">
  <strong>Task:</strong> : Train the model created in the previous cell on CIFAR10.
  <ul>
  <li>Train the network using `epochs = 30`</li>
  <li>Train the network using a `batch_size = 128`</li>
  <li>Use `validation_split = 0.2` when calling `fit()`</li>
  <li>Plot the losses and accuracy</li>
  <li>Assess the performance on the test set</li>
</ul>
We recommend that you do not copy-paste from the cells above, but rather re-write the code yourself. You can always take a look at earlier cells if you are unsure about what to do.
</div>
End of explanation
<END_TASK>