<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 TP1 - Handwritten digit recognition using neural networks and convolutional neural networks
M4108C/M4109C - INFOgr2D
03/2018
1. Introduction
In this lab, we first design and observe the performance of fully connected neural networks (NNs) as well as convolutional neural networks (CNNs) for digit regconition.
This lab uses the MNIST dataset and Keras implementation with Tensorflow backend.
Here are some useful links that may help you during the lab.
For NNs
- (https
Step1:  2) Display some image samples using matplotlib.pyplot
Step2:  3) (If necessary) Reduce the number of training images (for quick training and small-GPU computer)
Step3:  3. Fully-connected NNs
1) Preprocess our data
- In order to perform fully-connected NNs, we need to first reshape the image (size of 28x28 pixels) to a 1D array (usingthe reshape function)
Step4:  Convert the data to float and scale the values between 0 to 1
Convert the labels from integer to categorical
Step5:  2) Define a 2-layer fully connected NN with 256 neurons for the 1st layer and 512 neurons for the 2nd layer. Use the 'sigmoid' activation function for the 2 network layers and 'softmax' for the final output layer. Show the model summary.
Step6:  3) Compile and train the model. Report the training time
Step7:  4) test the model
Step8:  4. CNNs
1) Now, start to design your first CNN (with one CONV layer followed by one MAX-POOLING + 1 FC layer). 
- CONV layer
Step9:  4) Now, add one more CONV layer followed by on Max pooling. The number of filters and filter size to be chosen by yourself. 
Report the performance. Report the performance. Show the number of parameters as well as computational time.
Step10:  5. CNNs (advanced)
Continue to design other CNNs by
<ASSISTANT_TASK:>
Python Code:
import numpy as np
import keras
from keras.datasets import mnist
# fix random seed for reproducibility
np.random.seed(7)
# load data
(x_train,y_train), (x_test,y_test) = mnist.load_data()
# show the training size, test size, number of class
print("train size : ",x_train.shape)
print("test size : ",x_test.shape)
print("train label : ",y_train.shape)
print("test label : ",y_test.shape)
nclass = len(np.unique(y_train))
print("number of classes:",nclass)
Explanation: TP1 - Handwritten digit recognition using neural networks and convolutional neural networks
M4108C/M4109C - INFOgr2D
03/2018
1. Introduction
In this lab, we first design and observe the performance of fully connected neural networks (NNs) as well as convolutional neural networks (CNNs) for digit regconition.
This lab uses the MNIST dataset and Keras implementation with Tensorflow backend.
Here are some useful links that may help you during the lab.
For NNs
- (https://www.learnopencv.com/image-classification-using-feedforward-neural-network-in-keras/)
- (https://ankivil.com/mnist-database-and-simple-classification-networks/)
For CNNs
- (https://sefiks.com/2017/11/05/handwritten-digit-recognition-using-cnn-with-keras/)
- (http://adventuresinmachinelearning.com/keras-tutorial-cnn-11-lines/)
2. MNIST dataset
1) Load MNIST dataset and show its training and test sizes
End of explanation
%matplotlib inline
import matplotlib.pyplot as plt
# to complete
for i in range(0,9):
    plt.subplot(3, 3, i+1)
    plt.imshow(x_train[i], cmap=plt.get_cmap('gray')); plt.axis('off')
Explanation: 2) Display some image samples using matplotlib.pyplot
End of explanation
#x_train = x_train[0:30000,:]
#y_train = y_train[0:30000]
print("train size : ",x_train.shape)
print("train label : ",y_train.shape)
Explanation: 3) (If necessary) Reduce the number of training images (for quick training and small-GPU computer)
End of explanation
x_train = x_train.reshape(x_train.shape[0], 28*28)
x_test = x_test.reshape(x_test.shape[0], 28*28)
print("train size : ",x_train.shape)
print("test size : ",x_test.shape)
Explanation: 3. Fully-connected NNs
1) Preprocess our data
- In order to perform fully-connected NNs, we need to first reshape the image (size of 28x28 pixels) to a 1D array (usingthe reshape function)
End of explanation
x_train = x_train.astype('float32')/255
x_test = x_test.astype('float32')/255
from keras.utils import np_utils
y_train_cat = np_utils.to_categorical(y_train, nclass)
y_test_cat = np_utils.to_categorical(y_test, nclass)
y_train_cat.shape
Explanation: Convert the data to float and scale the values between 0 to 1
Convert the labels from integer to categorical
End of explanation
from keras.models import Sequential
from keras.layers.core import Dense, Dropout, Activation
from keras.optimizers import RMSprop
# define the model
model_nn = Sequential()
model_nn.add(Dense(256, input_shape=(784,),activation='sigmoid'))
model_nn.add(Dense(512, activation='sigmoid'))
model_nn.add(Dense(10,activation='softmax')) #Last layer has nclass nodes
model_nn.summary()
Explanation: 2) Define a 2-layer fully connected NN with 256 neurons for the 1st layer and 512 neurons for the 2nd layer. Use the 'sigmoid' activation function for the 2 network layers and 'softmax' for the final output layer. Show the model summary.
End of explanation
import time
# compile the model
model_nn.compile(loss='categorical_crossentropy', optimizer =RMSprop(lr=0.001), metrics=["accuracy"])
# train the model
start_t_mod= time.time()
model_nn.fit(x_train, y_train_cat, batch_size=500, epochs = 20)
finish_t_mod = time.time()
time = finish_t_mod - start_t_mod
print("training time :", time)
Explanation: 3) Compile and train the model. Report the training time
End of explanation
# test the model
# 
score = model_nn.evaluate(x_test, y_test_cat)
print('Test loss:', score[0])
print('Test accuracy:', score[1])
Explanation: 4) test the model
End of explanation
from keras.models import Sequential
from keras.layers import Conv2D
from keras.layers import MaxPooling2D
from keras.layers import Dense, Activation, Flatten
import time
(x_train,y_train), (x_test,y_test) = mnist.load_data()
x_train = x_train.reshape(x_train.shape[0],28,28,1)
x_test = x_test.reshape(x_test.shape[0],28,28,1)
x_train = x_train.astype('float32')/255
x_test = x_test.astype('float32')/255
y_train_cat = np_utils.to_categorical(y_train, nclass)
y_test_cat = np_utils.to_categorical(y_test, nclass)
# design the model
model_cnn = Sequential()
model_cnn.add(Conv2D(32,(5,5),input_shape=(28,28,1), activation='relu'))
model_cnn.add(MaxPooling2D(pool_size=(2,2)))
model_cnn.add(Flatten())
model_cnn.add(Dense(128,activation='relu'))
model_cnn.add(Dense(10,activation='softmax'))
model_cnn.summary()
# compile the model
model_cnn.compile(loss='categorical_crossentropy',metrics=['accuracy'],optimizer=RMSprop(lr=0.001))
# train the model
start_t= time.time()
model_cnn.fit(x_train, y_train_cat, batch_size=128, epochs=5)
finish_t = time.time()
time_cnn = finish_t - start_t
print("training time :", time_cnn)
# test the model
score = model_cnn.evaluate(x_test, y_test_cat)
print('Test loss:', score[0])
print('Test accuracy:', score[1])
Explanation: 4. CNNs
1) Now, start to design your first CNN (with one CONV layer followed by one MAX-POOLING + 1 FC layer). 
- CONV layer: 32 filters of size: 5x5, with padding, no stride (stride = 1), activation 'Relu'
- MAX-POOLING 2x2
- FC layer 128 nodes, activation Relu
- Output dense layer with activation 'softmax'
(Do not forget your activation function)
Report the performance. Show the number of parameters as well as computational time.
End of explanation
# design new model
# design the model
model_cnn = Sequential()
model_cnn.add(Conv2D(32,(5,5),input_shape=(28,28,1), activation='relu'))
model_cnn.add(MaxPooling2D(pool_size=(2,2)))
model_cnn.add(Conv2D(32,(5,5), activation='relu'))
model_cnn.add(MaxPooling2D(pool_size=(2,2)))
model_cnn.add(Flatten())
model_cnn.add(Dense(128,activation='relu'))
model_cnn.add(Dense(10,activation='softmax'))
model_cnn.summary()
# compile the model
# train the model
# test the model
Explanation: 4) Now, add one more CONV layer followed by on Max pooling. The number of filters and filter size to be chosen by yourself. 
Report the performance. Report the performance. Show the number of parameters as well as computational time.
End of explanation
# Batch normalization
from keras.layers import BatchNormalization
# Drop out
from keras.layers import Dropout
# design new model
Explanation: 5. CNNs (advanced)
Continue to design other CNNs by:
- adding more layers (conv layer, FC layer)? 
- using Batch normalization? 
- using Dropout? 
- sing Data augmentation?
Report the performance.
End of explanation
<END_TASK>