<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 Deep Learning
Assignment 4
Previously in 2_fullyconnected.ipynb and 3_regularization.ipynb, we trained fully connected networks to classify notMNIST characters.
The goal of this assignment is make the neural network convolutional.
Step1:  Reformat into a TensorFlow-friendly shape
Step2:  Let's build a small network with two convolutional layers, followed by one fully connected layer. Convolutional networks are more expensive computationally, so we'll limit its depth and number of fully connected nodes.
Step3:  Problem 1
The convolutional model above uses convolutions with stride 2 to reduce the dimensionality. Replace the strides by a max pooling operation (nn.max_pool()) of stride 2 and kernel size 2.
Step4:  Output
Initialized
Minibatch loss at step 0
<ASSISTANT_TASK:>
Python Code:
# These are all the modules we'll be using later. Make sure you can import them
# before proceeding further.
from __future__ import print_function
import numpy as np
import tensorflow as tf
from six.moves import cPickle as pickle
from six.moves import range
pickle_file = '../notMNIST.pickle'
with open(pickle_file, 'rb') as f:
  save = pickle.load(f)
  train_dataset = save['train_dataset']
  train_labels = save['train_labels']
  valid_dataset = save['valid_dataset']
  valid_labels = save['valid_labels']
  test_dataset = save['test_dataset']
  test_labels = save['test_labels']
  del save  # hint to help gc free up memory
  print('Training set', train_dataset.shape, train_labels.shape)
  print('Validation set', valid_dataset.shape, valid_labels.shape)
  print('Test set', test_dataset.shape, test_labels.shape)
Explanation: Deep Learning
Assignment 4
Previously in 2_fullyconnected.ipynb and 3_regularization.ipynb, we trained fully connected networks to classify notMNIST characters.
The goal of this assignment is make the neural network convolutional.
End of explanation
image_size = 28
num_labels = 10
num_channels = 1 # grayscale
import numpy as np
def reformat(dataset, labels):
  dataset = dataset.reshape(
    (-1, image_size, image_size, num_channels)).astype(np.float32)
  labels = (np.arange(num_labels) == labels[:,None]).astype(np.float32)
  return dataset, labels
train_dataset, train_labels = reformat(train_dataset, train_labels)
valid_dataset, valid_labels = reformat(valid_dataset, valid_labels)
test_dataset, test_labels = reformat(test_dataset, test_labels)
print('Training set', train_dataset.shape, train_labels.shape)
print('Validation set', valid_dataset.shape, valid_labels.shape)
print('Test set', test_dataset.shape, test_labels.shape)
def accuracy(predictions, labels):
  return (100.0 * np.sum(np.argmax(predictions, 1) == np.argmax(labels, 1))
          / predictions.shape[0])
Explanation: Reformat into a TensorFlow-friendly shape:
- convolutions need the image data formatted as a cube (width by height by #channels)
- labels as float 1-hot encodings.
End of explanation
batch_size = 16
patch_size = 5
depth = 16
num_hidden = 64
graph = tf.Graph()
with graph.as_default():
  # Input data.
  tf_train_dataset = tf.placeholder(
    tf.float32, shape=(batch_size, image_size, image_size, num_channels))
  tf_train_labels = tf.placeholder(tf.float32, shape=(batch_size, num_labels))
  tf_valid_dataset = tf.constant(valid_dataset)
  tf_test_dataset = tf.constant(test_dataset)
  
  # Variables.
  layer1_weights = tf.Variable(tf.truncated_normal(
      [patch_size, patch_size, num_channels, depth], stddev=0.1))
  layer1_biases = tf.Variable(tf.zeros([depth]))
  layer2_weights = tf.Variable(tf.truncated_normal(
      [patch_size, patch_size, depth, depth], stddev=0.1))
  layer2_biases = tf.Variable(tf.constant(1.0, shape=[depth]))
  layer3_weights = tf.Variable(tf.truncated_normal(
      [image_size // 4 * image_size // 4 * depth, num_hidden], stddev=0.1))
  layer3_biases = tf.Variable(tf.constant(1.0, shape=[num_hidden]))
  layer4_weights = tf.Variable(tf.truncated_normal(
      [num_hidden, num_labels], stddev=0.1))
  layer4_biases = tf.Variable(tf.constant(1.0, shape=[num_labels]))
  
  # Model.
  def model(data): #data of shape [batch_size, image_size, image_size, num_channels]
    conv = tf.nn.conv2d(data, layer1_weights, [1, 2, 2, 1], padding='SAME') # shape of [batch_size, image_size/2, image_size/2, depth]
    hidden = tf.nn.relu(conv + layer1_biases)
    conv = tf.nn.conv2d(hidden, layer2_weights, [1, 2, 2, 1], padding='SAME')# shape of [batch_size, image_size/4, image_size/4, depth]
    hidden = tf.nn.relu(conv + layer2_biases)
    shape = hidden.get_shape().as_list()
    reshape = tf.reshape(hidden, [shape[0], shape[1] * shape[2] * shape[3]])# shape of [batch_size, image_size/4 * image_size/4* depth]
    hidden = tf.nn.relu(tf.matmul(reshape, layer3_weights) + layer3_biases) # shape of [batch_size,num_hidden]
    return tf.matmul(hidden, layer4_weights) + layer4_biases # shape of [batch_size,num_labels]
  
  # Training computation.
  logits = model(tf_train_dataset)
  loss = tf.reduce_mean(
    tf.nn.softmax_cross_entropy_with_logits(labels=tf_train_labels, logits=logits))
    
  # Optimizer.
  optimizer = tf.train.GradientDescentOptimizer(0.05).minimize(loss)
  
  # Predictions for the training, validation, and test data.
  train_prediction = tf.nn.softmax(logits)
  valid_prediction = tf.nn.softmax(model(tf_valid_dataset))
  test_prediction = tf.nn.softmax(model(tf_test_dataset))
num_steps = 1001
with tf.Session(graph=graph) as session:
  tf.global_variables_initializer().run()
  print('Initialized')
  for step in range(num_steps):
    offset = (step * batch_size) % (train_labels.shape[0] - batch_size)
    batch_data = train_dataset[offset:(offset + batch_size), :, :, :]
    batch_labels = train_labels[offset:(offset + batch_size), :]
    feed_dict = {tf_train_dataset : batch_data, tf_train_labels : batch_labels}
    _, l, predictions = session.run(
      [optimizer, loss, train_prediction], feed_dict=feed_dict)
    if (step % 50 == 0):
      print('Minibatch loss at step %d: %f' % (step, l))
      print('Minibatch accuracy: %.1f%%' % accuracy(predictions, batch_labels))
      print('Validation accuracy: %.1f%%' % accuracy(
        valid_prediction.eval(), valid_labels))
  print('Test accuracy: %.1f%%' % accuracy(test_prediction.eval(), test_labels))
Explanation: Let's build a small network with two convolutional layers, followed by one fully connected layer. Convolutional networks are more expensive computationally, so we'll limit its depth and number of fully connected nodes.
End of explanation
  # Variables.
  layer3_weights = tf.Variable(tf.truncated_normal(
      [image_size // 2 * image_size // 2 * num_channels, num_hidden], stddev=0.1))
  layer3_biases = tf.Variable(tf.constant(1.0, shape=[num_hidden]))
  layer4_weights = tf.Variable(tf.truncated_normal(
      [num_hidden, num_labels], stddev=0.1))
  layer4_biases = tf.Variable(tf.constant(1.0, shape=[num_labels]))
  
  # Model.
  def model(data):
    # Data is shaped of [batch_size, image_size, image_size, num_channels]
    hidden = tf.nn.max_pool(data, [1, 2, 2, 1],[1, 2, 2, 1] , padding='SAME') #same shape of [batch_size, image_size/2, image_size/2, num_channels]
    shape = hidden.get_shape().as_list() 
    reshape = tf.reshape(hidden, [shape[0], shape[1] * shape[2] * shape[3]]) #reshaped into 2D array of [batch_size, image_size/2* image_size/2 * num_channels]
    hidden = tf.nn.relu(tf.matmul(reshape, layer3_weights) + layer3_biases)
    return tf.matmul(hidden, layer4_weights) + layer4_biases
Explanation: Problem 1
The convolutional model above uses convolutions with stride 2 to reduce the dimensionality. Replace the strides by a max pooling operation (nn.max_pool()) of stride 2 and kernel size 2.
End of explanation
  # Variables.
  layer1_weights = tf.Variable(tf.truncated_normal(
      [patch_size, patch_size, num_channels, depth], stddev=0.1))
  layer1_biases = tf.Variable(tf.zeros([depth]))
  layer2_weights = tf.Variable(tf.truncated_normal(
      [patch_size, patch_size, depth, depth], stddev=0.1))
  layer2_biases = tf.Variable(tf.constant(1.0, shape=[depth]))
  layer3_weights = tf.Variable(tf.truncated_normal(
      [math.ceil(image_size / 16)  * math.ceil(image_size /16) * depth, num_hidden], stddev=0.1))
  layer3_biases = tf.Variable(tf.constant(1.0, shape=[num_hidden]))
  layer4_weights = tf.Variable(tf.truncated_normal(
      [num_hidden, num_labels], stddev=0.1))
  layer4_biases = tf.Variable(tf.constant(1.0, shape=[num_labels]))
  
  # Model.
  def model(data):
    conv = tf.nn.conv2d(data, layer1_weights, [1, 2, 2, 1], padding='SAME') # shape of [batch_size, image_size/2, image_size/2, depth]: [16, 14, 14, 16]
    shape1 = conv.get_shape().as_list()
    hidden = tf.nn.relu(conv + layer1_biases)
    hidden1 = tf.nn.max_pool(hidden, [1, 2, 2, 1], [1, 2, 2, 1] , padding='SAME') #shape of [batch_size, image_size/4, image_size/4, depth]: [16, 7, 7, 16]
    shape2 = hidden1.get_shape().as_list()
    conv = tf.nn.conv2d(hidden1, layer2_weights, [1, 2, 2, 1], padding='SAME') #shape of [batch_size, image_size/8, image_size/8, depth]: [16, 4, 4, 16]
    shape3 = conv.get_shape().as_list()
    hidden = tf.nn.relu(conv + layer2_biases)
    hidden2 = tf.nn.max_pool(hidden, [1, 2, 2, 1],[1, 2, 2, 1] , padding='SAME') #same shape of [batch_size, image_size/16, image_size/16, depth]: [16, 2, 2, 16]
    shape = hidden2.get_shape().as_list()
    reshape = tf.reshape(hidden2, [shape[0], shape[1] * shape[2] * shape[3]])
    hidden = tf.nn.relu(tf.matmul(reshape, layer3_weights) + layer3_biases)
    return tf.matmul(hidden, layer4_weights) + layer4_biases
Explanation: Output
Initialized
Minibatch loss at step 0: 2.694045
Minibatch accuracy: 0.0%
Validation accuracy: 7.9%
Minibatch loss at step 50: 1.684043
Minibatch accuracy: 50.0%
Validation accuracy: 65.4%
Minibatch loss at step 100: 0.953076
Minibatch accuracy: 81.2%
Validation accuracy: 70.7%
Minibatch loss at step 150: 0.509777
Minibatch accuracy: 87.5%
Validation accuracy: 76.1%
Minibatch loss at step 200: 0.464738
Minibatch accuracy: 87.5%
Validation accuracy: 78.0%
Minibatch loss at step 250: 0.875403
Minibatch accuracy: 75.0%
Validation accuracy: 78.1%
Minibatch loss at step 300: 0.913955
Minibatch accuracy: 75.0%
Validation accuracy: 80.6%
Minibatch loss at step 350: 0.677745
Minibatch accuracy: 75.0%
Validation accuracy: 80.3%
Minibatch loss at step 400: 0.772082
Minibatch accuracy: 68.8%
Validation accuracy: 80.2%
Minibatch loss at step 450: 0.742608
Minibatch accuracy: 93.8%
Validation accuracy: 81.0%
Minibatch loss at step 500: 0.643120
Minibatch accuracy: 81.2%
Validation accuracy: 80.2%
Minibatch loss at step 550: 0.628266
Minibatch accuracy: 75.0%
Validation accuracy: 80.5%
Minibatch loss at step 600: 0.979889
Minibatch accuracy: 81.2%
Validation accuracy: 80.8%
Minibatch loss at step 650: 0.581403
Minibatch accuracy: 87.5%
Validation accuracy: 81.0%
Minibatch loss at step 700: 0.751648
Minibatch accuracy: 75.0%
Validation accuracy: 80.5%
Minibatch loss at step 750: 0.344367
Minibatch accuracy: 87.5%
Validation accuracy: 81.4%
Minibatch loss at step 800: 0.698404
Minibatch accuracy: 87.5%
Validation accuracy: 81.8%
Minibatch loss at step 850: 0.795159
Minibatch accuracy: 68.8%
Validation accuracy: 81.2%
Minibatch loss at step 900: 0.890547
Minibatch accuracy: 75.0%
Validation accuracy: 80.8%
Minibatch loss at step 950: 1.138554
Minibatch accuracy: 75.0%
Validation accuracy: 81.2%
Minibatch loss at step 1000: 0.733926
Minibatch accuracy: 81.2%
Validation accuracy: 81.4%
Test accuracy: 87.4%
We can see that the max_pool on its own is not too bad at all, without any convolution and it is much faster to compute.
Problem 2
Try to get the best performance you can using a convolutional net. Look for example at the classic LeNet5 architecture, adding Dropout, and/or adding learning rate decay.
2.1 Simple LeNet Idea
Conv2D + max-pool + Conv2D + max-pool + full-network + output
End of explanation
<END_TASK>