<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 MNIST classification
In this session we will learn to design and train simple neural networks and observe their performance on a digit dataset called MNIST. To do this, we will use the Keras API with tensorflow, which will allow us to train and evaluate models in a simple way with a few lines of code.
Step1:  Dataset
Let's begin by loading the MNIST dataset, which we will use during the whole session.
Step2:  Multiclass softmax
We will design a network with a single layer, with as many neurons as categories in our dataset. Each output will be a fuction of all inputs (pixels)
Step3:  Exercise
Step4:  Categories need to be converted to one-hot vectors for training
Step5:  We are now ready to train. Let's define the optimizer
Step6:  In Keras, we need to compile the model to define the loss and the optimizer we want to use. Since we are dealing with a classification problem, we will use the cross entropy loss, which is already defined in keras. Additionally, we will incorporate the accuracy as an additional metric to compute at the end of each epoch
Step7:  Now let's train the model. model.fit() will do the training loop for us. We just need to pass the training data X_train and labels Y_train as input, specify the batch_size and the number of epochs nb_epoch we want to do. We also pass the test set (X_test,Y_Test) as validation data, which will allow us to see how the model performs on the test data as training progresses. Let's run it
Step8:  We can plot the loss and accuracy curves with the history object returned by model.fit(). The function plot_curves, which is defined in utils.py will do this for us.
Step9:  The curve trend indicates that the model may be able to improve if we train it for longer, but for now let's leave it here.
Let's now evaluate our model. model.evaluate() will take all the test samples, forward them through the network and return the average loss, and any additional metrics we specified (in our case, the accuracy).
Step10:  Adding a hidden layer
Let's try to train a model with a hidden layer between the input and the classifier. 
Exercise
Step11:  Exercise
Step12:  Dropout
Exercise
<ASSISTANT_TASK:>
Python Code:
import matplotlib.pyplot as plt  
%matplotlib inline
from utils import plot_samples, plot_curves
import time
import numpy as np
# force random seed for results to be reproducible
SEED = 4242
np.random.seed(SEED)
Explanation: MNIST classification
In this session we will learn to design and train simple neural networks and observe their performance on a digit dataset called MNIST. To do this, we will use the Keras API with tensorflow, which will allow us to train and evaluate models in a simple way with a few lines of code.
End of explanation
from keras.datasets import mnist
from keras.utils import np_utils
# Load pre-shuffled MNIST data into train and test sets
(X_train, y_train), (X_test, y_test) = mnist.load_data()
# Display some of the samples
plot_samples(X_train)
X_train.shape
Explanation: Dataset
Let's begin by loading the MNIST dataset, which we will use during the whole session.
End of explanation
from keras.models import Sequential
from keras.layers import Dense, Activation
model = Sequential()
# in the first layer we need to specify the input shape
model.add(Dense(10, input_shape=(784,)))
model.add(Activation('softmax'))
model.summary()
Explanation: Multiclass softmax
We will design a network with a single layer, with as many neurons as categories in our dataset. Each output will be a fuction of all inputs (pixels)
End of explanation
X_train = X_train.reshape(60000, 784)
X_test = X_test.reshape(10000, 784)
X_train = X_train.astype('float32')
X_test = X_test.astype('float32')
X_train /= 255
X_test /= 255
Explanation: Exercise: model.summary() gave us the total number of trainable parameters of our model. How is this number obtained? 
We flatten and normalize images to match the input that the network expects:
End of explanation
nb_classes = 10
Y_train = np_utils.to_categorical(y_train, nb_classes)
Y_test = np_utils.to_categorical(y_test, nb_classes)
y_train, Y_train
Explanation: Categories need to be converted to one-hot vectors for training:
End of explanation
from keras.optimizers import SGD
lr = 0.01
# For now we will not decrease the learning rate
decay = 0
optim = SGD(lr=lr, decay=decay, momentum=0.9, nesterov=True)
Explanation: We are now ready to train. Let's define the optimizer:
End of explanation
model.compile(loss='categorical_crossentropy',
              optimizer=optim,
              metrics=['accuracy'])
Explanation: In Keras, we need to compile the model to define the loss and the optimizer we want to use. Since we are dealing with a classification problem, we will use the cross entropy loss, which is already defined in keras. Additionally, we will incorporate the accuracy as an additional metric to compute at the end of each epoch:
End of explanation
batch_size = 32
nb_epoch = 20
verbose = 2
t = time.time()
history = model.fit(X_train, Y_train,
                batch_size=batch_size, nb_epoch=nb_epoch,
                verbose=verbose,validation_data=(X_test, Y_test))
print (time.time() - t, "seconds.")
Explanation: Now let's train the model. model.fit() will do the training loop for us. We just need to pass the training data X_train and labels Y_train as input, specify the batch_size and the number of epochs nb_epoch we want to do. We also pass the test set (X_test,Y_Test) as validation data, which will allow us to see how the model performs on the test data as training progresses. Let's run it:
End of explanation
plot_curves(history,nb_epoch)
Explanation: We can plot the loss and accuracy curves with the history object returned by model.fit(). The function plot_curves, which is defined in utils.py will do this for us.
End of explanation
score = model.evaluate(X_test, Y_test, verbose=0)
print ("Loss: %f"%(score[0]))
print ("Accuracy: %f"%(score[1]))
Explanation: The curve trend indicates that the model may be able to improve if we train it for longer, but for now let's leave it here.
Let's now evaluate our model. model.evaluate() will take all the test samples, forward them through the network and return the average loss, and any additional metrics we specified (in our case, the accuracy).
End of explanation
import numpy as np
np.random.seed(SEED)
# MODEL DEFINITION
model = Sequential()
# ...
model.summary()
Explanation: Adding a hidden layer
Let's try to train a model with a hidden layer between the input and the classifier. 
Exercise: Modify the previous architecture to include this layer with 128 neurons and train it. Take into account that the input_shape must be passed to the first layer of the network.
End of explanation
# COMPILE & TRAIN
Explanation: Exercise: Compute the number of parameters and check if they match the ones given by model.summary()
Exercise: Write the code to train the model. Check the code we used for the previous model. Should be too similar...
End of explanation
import numpy as np
np.random.seed(SEED)
from keras.layers import Dropout
dratio = 0.2
H_DIM = 128
model = Sequential()
# ...
model.summary()
model.compile(loss='categorical_crossentropy',
              optimizer=optim,
              metrics=['accuracy'])
t = time.time()
history = model.fit(X_train, Y_train,
                batch_size=batch_size, nb_epoch=nb_epoch,
                verbose=verbose,validation_data=(X_test, Y_test))
print (time.time() - t, "seconds.")
score = model.evaluate(X_test, Y_test, verbose=0)
print ("-"*10)
print ("Loss: %f"%(score[0]))
print ("Accuracy: %f"%(score[1]))
plot_curves(history,nb_epoch)
Explanation: Dropout
Exercise: Add a dropout layer to the model and see their effect in the training curves & accuracy. See the documentation for the Dropout layer in keras.
End of explanation
<END_TASK>