<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 Introduction to Keras with MNIST
Import various modules that we need for this notebook.
Step1:  Load the MNIST dataset, flatten the images, convert the class labels, and scale the data.
Step2:  I. Basic example
Build and compile a basic model.
Step3:  Fit the model over 25 epochs.
Step4:  Evaluate model on the test set
Step5:  Predict classes on the test set.
Step6:  II. Deeper model with dropout and cross entropy
Let's now build a deeper model, with four hidden dense layers and dropout layers. I'll use rectified linear units as they tend to perform better on deep models. I also initilize the nodes using "glorot_normal", which uses Gaussian noise scaled by the sum of the inputs plus outputs from the node. Notice that we do not need to give an input shape to any layers other than the first.
Step7:  III. Small model
Step8:  The classification rate on the validation set is not nearly as predictive, but it is still not too bad overall. A model object contains a list of its layers. The weights are easy to pull out.
Step9:  The first set of weights will be given as weights the same size as the input space. Notice how
Step10:  The second layer of weights will be given as a single 16x16 matrix of weights.
Step11:  IV. Further tweaks
<ASSISTANT_TASK:>
Python Code:
%pylab inline
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
from keras.datasets import mnist
from keras.models import Sequential
from keras.layers.core import Dense, Dropout, Activation
from keras.optimizers import SGD, RMSprop
from keras.utils import np_utils
from keras.regularizers import l2
Explanation: Introduction to Keras with MNIST
Import various modules that we need for this notebook.
End of explanation
(X_train, y_train), (X_test, y_test) = mnist.load_data()
X_train = X_train.reshape(60000, 28**2).astype('float32') / 255
X_test = X_test.reshape(10000, 28**2).astype('float32') / 255
Y_train = np_utils.to_categorical(y_train, 10)
Y_test = np_utils.to_categorical(y_test, 10)
Explanation: Load the MNIST dataset, flatten the images, convert the class labels, and scale the data.
End of explanation
model = Sequential()
model.add(Dense(512, input_shape=(28 * 28,)))
model.add(Activation("sigmoid"))
model.add(Dense(10))
          
sgd = SGD(lr = 0.01, momentum = 0.9, nesterov = True)
model.compile(loss='mse', optimizer=sgd)
Explanation: I. Basic example
Build and compile a basic model.
End of explanation
model.fit(X_train, Y_train, batch_size=32, nb_epoch=10,
          verbose=1, show_accuracy=True, validation_split=0.1)
Explanation: Fit the model over 25 epochs.
End of explanation
print("Test classification rate %0.05f" % model.evaluate(X_test, Y_test, show_accuracy=True)[1])
Explanation: Evaluate model on the test set
End of explanation
y_hat = model.predict_classes(X_test)
pd.crosstab(y_hat, y_test)
Explanation: Predict classes on the test set.
End of explanation
model = Sequential()
model.add(Dense(512, input_shape=(28 * 28,), init="glorot_normal"))
model.add(Activation("relu"))
model.add(Dropout(0.5))
model.add(Dense(512, init="glorot_normal"))
model.add(Activation("relu"))
model.add(Dropout(0.5))
model.add(Dense(512, init="glorot_normal"))
model.add(Activation("relu"))
model.add(Dropout(0.5))
model.add(Dense(512, init="glorot_normal"))
model.add(Activation("relu"))
model.add(Dropout(0.5))
model.add(Dense(10))
model.add(Activation('softmax'))
sgd = SGD(lr = 0.01, momentum = 0.9, nesterov = True)
model.compile(loss='categorical_crossentropy', optimizer=sgd)
model.fit(X_train, Y_train, batch_size=32, nb_epoch=10,
          verbose=1, show_accuracy=True, validation_split=0.1)
print("Test classification rate %0.05f" % model.evaluate(X_test, Y_test, show_accuracy=True)[1])
fy_hat = model.predict_classes(X_test)
pd.crosstab(y_hat, y_test)
test_wrong = [im for im in zip(X_test,y_hat,y_test) if im[1] != im[2]]
plt.figure(figsize=(15, 15))
for ind, val in enumerate(test_wrong[:100]):
    plt.subplot(10, 10, ind + 1)
    im = 1 - val[0].reshape((28,28))
    axis("off")
    plt.imshow(im, cmap='gray')
Explanation: II. Deeper model with dropout and cross entropy
Let's now build a deeper model, with four hidden dense layers and dropout layers. I'll use rectified linear units as they tend to perform better on deep models. I also initilize the nodes using "glorot_normal", which uses Gaussian noise scaled by the sum of the inputs plus outputs from the node. Notice that we do not need to give an input shape to any layers other than the first.
End of explanation
model = Sequential()
model.add(Dense(16, input_shape=(28 * 28,), init="glorot_normal"))
model.add(Activation("relu"))
model.add(Dropout(0.5))
model.add(Dense(16, init="glorot_normal"))
model.add(Activation("relu"))
model.add(Dropout(0.5))
model.add(Dense(10))
model.add(Activation('softmax'))
rms = RMSprop()
model.compile(loss='categorical_crossentropy', optimizer=rms)
model.fit(X_train, Y_train, batch_size=32, nb_epoch=10,
          verbose=1, show_accuracy=True, validation_split=0.1)
Explanation: III. Small model: Visualizing weights
Now, I want to make a model that has only a small number of hidden nodes in each layer. We may then have a chance of actually visualizing the weights.
End of explanation
print(model.layers) # list of the layers
print(model.layers[0].get_weights()[0].shape) # the weights
Explanation: The classification rate on the validation set is not nearly as predictive, but it is still not too bad overall. A model object contains a list of its layers. The weights are easy to pull out.
End of explanation
W1 = model.layers[0].get_weights()[0]
for ind, val in enumerate(W1.T):
    plt.figure(figsize=(3, 3), frameon=False)
    im = val.reshape((28,28))
    plt.axis("off")
    plt.imshow(im, cmap='seismic')
Explanation: The first set of weights will be given as weights the same size as the input space. Notice how
End of explanation
W2 = model.layers[3].get_weights()[0]
plt.figure(figsize=(3, 3))
im = W2.reshape((16,16))
plt.axis("off")
plt.imshow(im, cmap='seismic')
Explanation: The second layer of weights will be given as a single 16x16 matrix of weights.
End of explanation
model = Sequential()
model.add(Dense(128, input_shape=(28 * 28,), init="glorot_normal"))
model.add(Activation("relu"))
model.add(Dropout(0.5))
model.add(Dense(512, init="glorot_normal",W_regularizer=l2(0.1)))
model.add(Activation("relu"))
model.add(Dropout(0.2))
model.add(Dense(512, init="glorot_normal",W_regularizer=l2(0.1)))
model.add(Activation("relu"))
model.add(Dropout(0.2))
model.add(Dense(10))
model.add(Activation('softmax'))
rms = RMSprop()
model.compile(loss='categorical_crossentropy', optimizer=rms)
model.fit(X_train, Y_train, batch_size=32, nb_epoch=5,
          verbose=1, show_accuracy=True, validation_split=0.1)
Explanation: IV. Further tweaks: weights and alternative optimizers
Just to show off a few more tweaks, we'll run one final model. Here we use weights and an alternative to vanillia stochastic gradient descent.
End of explanation
<END_TASK>