<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 MNIST classification
This lecture demonstrates an example of a classification instance using Tensorflow.
MNIST data
​
MNIST is a collection of images of handwritten numerical digits. Each image is 28x28 pixels and each has a grayscale value from 0-255.
Step1:  We have 60000 28x28 images in our training set and 10000 in the test set.
Step2:  The pixel value range is 0-255
Step3:  Let's see what they look like
Step4:  The dataset also includes a label for each digit
Step5:  Pre-processing
We will flatten each digit into a 784-dimensional vector.
Step6:  We will normalize the data to the [0,1] range
Step7:  We also need to convert the labels into a 1-hot representation
Step8:  Building a model
First let's import tensorflow
Step9:  The model will take the flattened digit as input. An input is declared as a "placeholder" variable meaning that the value of this tensor will be provided at run-time. For the computation of the loss function the class labels are also considered inputs to the model
Step10:  We will feed the input into a two dense layers with a tanh() non-linearity. Each layer consists of the weight matrix W and the bias vector b. These have to be declared as variables
Step11:  The activations of the layer are fed into a soft-max layer that outputs class probabilities
Step12:  We will use the cross-entropy between the predicted and the actual labels as our loss function
Step13:  We also need to define what a training step looks like. The command below tells tensorflow to optimize the loss function using a Stochastic Gradient Descent (SGD) step with a learning rate of 0.5
Step14:  We need to create a session that will run on the computational graph
Step15:  We also need to initialize the variables of the model
Step16:  We can use the session to feed the input to the model and get the value of a specific node
Step17:  Notice how we did not provide the labels since these are not part of the slice of the computational graph for the class probabilities.
We can also perform an SGD step on this batch by running the session on the training step
Step18:  Notice how re-running the command above shifts the output of the model towards the actual labels. Of course this instance of the model will be horribly overfit to these few digits. Let's re-initialize the model
Step19:  Instead we will cycle over the whole training dataset a few times. We will process the dataset in mini-batches and take an SGD step for each such mini-batch.
Step20:  Let's see what the model learns
Step21:  To test the model more 'formally' we can compute the accuracy on the test dataset
<ASSISTANT_TASK:>
Python Code:
import numpy as np;
import matplotlib
import matplotlib.pyplot as plt
mnist = np.load('mnist_data.npz')
X_train = mnist['X_train']
X_test = mnist['X_test']
y_train = mnist['y_train']
y_test = mnist['y_test']
Explanation: MNIST classification
This lecture demonstrates an example of a classification instance using Tensorflow.
MNIST data
​
MNIST is a collection of images of handwritten numerical digits. Each image is 28x28 pixels and each has a grayscale value from 0-255.
End of explanation
print(X_train.shape)
print(X_test.shape)
Explanation: We have 60000 28x28 images in our training set and 10000 in the test set.
End of explanation
print(X_train[0])
plt.imshow(X_train[0],cmap='gray')
plt.axis('off')
plt.show()
Explanation: The pixel value range is 0-255
End of explanation
sz = 5;
for i in range(sz*sz):
    plt.subplot(sz, sz, i+1)
    plt.imshow(X_train[i],cmap='gray')
    plt.axis('off')
plt.show()
Explanation: Let's see what they look like:
End of explanation
print(y_train[:sz*sz].reshape([sz,sz]))
Explanation: The dataset also includes a label for each digit:
End of explanation
train_data = X_train.reshape([-1,28*28])
test_data = X_test.reshape([-1,28*28])
print(train_data.shape)
print(test_data.shape)
Explanation: Pre-processing
We will flatten each digit into a 784-dimensional vector.
End of explanation
def normalize_data(x):
	flat_x = np.reshape(x,[-1]);
	L = np.min(x);
	H = np.max(x);
    
	return (x.astype(np.float32)-L)/(H-L);
train_data = normalize_data(train_data)
test_data = normalize_data(test_data)
print(train_data[0])
Explanation: We will normalize the data to the [0,1] range:
End of explanation
def to_one_hot(labels,num):
    one_hot_labels = np.zeros((labels.shape[0],num))
    one_hot_labels[np.arange(labels.shape[0]),labels] = 1.0
    return one_hot_labels;
                              
train_labels = to_one_hot(y_train,10)
test_labels = to_one_hot(y_test,10)
print('Initial labels:')
print(y_train[:5])
print('1-hot representation:')
print(train_labels[:5])
Explanation: We also need to convert the labels into a 1-hot representation:
End of explanation
import tensorflow as tf
Explanation: Building a model
First let's import tensorflow:
End of explanation
x = tf.placeholder(tf.float32, [None, 784])
y = tf.placeholder(tf.float32, [None, 10])
Explanation: The model will take the flattened digit as input. An input is declared as a "placeholder" variable meaning that the value of this tensor will be provided at run-time. For the computation of the loss function the class labels are also considered inputs to the model:
End of explanation
h1_sz = 64;
W1 = tf.get_variable("W1", [784,h1_sz])
b1 = tf.Variable(tf.zeros([h1_sz]))
h1 = tf.matmul(x,W1) + b1
h1 = tf.tanh(h1)
W2 = tf.get_variable("W2", [h1_sz,10])
b2 = tf.Variable(tf.zeros([10]))
h2 = tf.matmul(h1,W2) + b2
Explanation: We will feed the input into a two dense layers with a tanh() non-linearity. Each layer consists of the weight matrix W and the bias vector b. These have to be declared as variables:
End of explanation
class_probs = tf.nn.softmax(h2)
Explanation: The activations of the layer are fed into a soft-max layer that outputs class probabilities:
End of explanation
cross_entropy_loss = tf.reduce_mean(-tf.reduce_sum(y * tf.log(tf.nn.softmax(class_probs)),axis=[1]))
Explanation: We will use the cross-entropy between the predicted and the actual labels as our loss function:
End of explanation
train_step = tf.train.GradientDescentOptimizer(0.5).minimize(cross_entropy_loss)
Explanation: We also need to define what a training step looks like. The command below tells tensorflow to optimize the loss function using a Stochastic Gradient Descent (SGD) step with a learning rate of 0.5:
End of explanation
sess = tf.InteractiveSession()
Explanation: We need to create a session that will run on the computational graph:
End of explanation
tf.global_variables_initializer().run()
Explanation: We also need to initialize the variables of the model:
End of explanation
batch_x = train_data[:5]
batch_y = train_labels[:5]
vis_probs = sess.run([class_probs], feed_dict={x:batch_x})
print(vis_probs)
Explanation: We can use the session to feed the input to the model and get the value of a specific node:
End of explanation
for j in range(200):
    [a,vis_probs] = sess.run([train_step,class_probs], feed_dict={x:batch_x,y:batch_y})
print(vis_probs)
Explanation: Notice how we did not provide the labels since these are not part of the slice of the computational graph for the class probabilities.
We can also perform an SGD step on this batch by running the session on the training step:
End of explanation
tf.global_variables_initializer().run()
Explanation: Notice how re-running the command above shifts the output of the model towards the actual labels. Of course this instance of the model will be horribly overfit to these few digits. Let's re-initialize the model:
End of explanation
epochs = 5;
batch_size = 32;
N = train_data.shape[0];
hist_loss = [];
for epoch in range(epochs):
	print("Epoch:", epoch)
	for index in range(int(N/(batch_size))):
        
		batch_x = train_data[index*batch_size:(index+1)*batch_size];
		batch_y = train_labels[index*batch_size:(index+1)*batch_size];
        
		[vis_loss,a] = sess.run([cross_entropy_loss,train_step], feed_dict={x:batch_x,y:batch_y})
		hist_loss += [vis_loss]
        
plt.plot(hist_loss)
plt.show()
Explanation: Instead we will cycle over the whole training dataset a few times. We will process the dataset in mini-batches and take an SGD step for each such mini-batch.
End of explanation
for j in range(5):
    digit = test_data[j].reshape([1,784]);
    actual_label = test_labels[j];
    
    plt.imshow(digit.reshape([28,28]),cmap='gray')
    plt.show()
    print(actual_label)
    
    [vis_probs] = sess.run([class_probs], feed_dict={x:digit})
    print(vis_probs)
    input()
Explanation: Let's see what the model learns:
End of explanation
correct_prediction = tf.equal(tf.argmax(y, 1), tf.argmax(class_probs, 1))
accuracy = tf.reduce_mean(tf.cast(correct_prediction, tf.float32))
train_accuracy = sess.run(accuracy,feed_dict={x:train_data,y:train_labels})
test_accuracy = sess.run(accuracy,feed_dict={x:test_data,y:test_labels})
print("Accuracy on training data:",train_accuracy)
print("Accuracy on test data:",test_accuracy)
Explanation: To test the model more 'formally' we can compute the accuracy on the test dataset:
End of explanation
<END_TASK>