<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 MNIST ML Pros tutorial
This notebook is based on the tutorial found here
This tutorial is very similar to the beginners tutorial except for some incremental improvements added to the end to improve the accuracy
Get the MNIST dataset
Step1:  Helper functions
Here we place our helper functions for creating weight & bias variables as well as doing our vanilla 2D convolution and Pooling operations
Step2:  First Convolution Layer
Our first layer consists of a convolution layer followed by a max pooling layer.  It will compute 32 features for each 5x5 patch.
The weight tensor has the shape
[patch_width, patch_height, num_input_channels, num_output_channels]
Reshape our image to a 4D tensor with the second and third dimension corresponding to image size and the final dimension for the number of colors.  The -1 in this case indicates the dimension that will be automatically modified to keep the size of the new tensor the same as the original.
Step3:  Second Convolution Layer
We create a similar structure except now we have 32 inputs and 64 feature outputs for each 5x5 patch.
Step4:  Densely Connected Layer
We have now done 2 2x2 convolutions which have reduced our image size to 7x7 since every 2x2 convolution effectively produces a new image that is half the size of the input image.
But for each 7x7 image, we now have 64 features.  So we will add a layer with 1024 neurons to allow processing on the entire image.
Step5:  Dropout Layer
The dropout layer helps to reduce overfitting by dropping connections between neurons in the densely connected layers.  This paper has a nice discussion on the matter.
Step6:  Readout Layer
We add a layer that takes the output of our fully connected layer and does a softmax regression into our classes.
Step7:  Training
It should not that depending on the CPU available this could take some time to complete.
<ASSISTANT_TASK:>
Python Code:
import tensorflow as tf
from tensorflow.examples.tutorials.mnist import input_data
mnist = input_data.read_data_sets("/data/MNIST/",one_hot=True)
sess = tf.InteractiveSession()
Explanation: MNIST ML Pros tutorial
This notebook is based on the tutorial found here
This tutorial is very similar to the beginners tutorial except for some incremental improvements added to the end to improve the accuracy
Get the MNIST dataset
End of explanation
def weight_variable(shape):
    initial = tf.truncated_normal(shape,stddev=0.1)
    return tf.Variable(initial)
    
def bias_variable(shape):
    initial = tf.constant(0.1, shape=shape)
    return tf.Variable(initial)
def conv2d(x,W):
    return tf.nn.conv2d(x,W,strides=[1,1,1,1],padding='SAME')
def max_pool_2x2(x):
    return tf.nn.max_pool(x,ksize=[1,2,2,1],strides=[1,2,2,1], padding='SAME')
# Setup our Input placeholder
x = tf.placeholder(tf.float32, [None, 784])
# Define loss and optimizer
y_ = tf.placeholder(tf.float32,[None,10])
Explanation: Helper functions
Here we place our helper functions for creating weight & bias variables as well as doing our vanilla 2D convolution and Pooling operations
End of explanation
W_conv1 = weight_variable([5,5,1,32])
b_conv1 = bias_variable([32])
x_image = tf.reshape(x,[-1,28,28,1])
h_conv1 = tf.nn.relu(conv2d(x_image,W_conv1) + b_conv1)
h_pool1 = max_pool_2x2(h_conv1)
Explanation: First Convolution Layer
Our first layer consists of a convolution layer followed by a max pooling layer.  It will compute 32 features for each 5x5 patch.
The weight tensor has the shape
[patch_width, patch_height, num_input_channels, num_output_channels]
Reshape our image to a 4D tensor with the second and third dimension corresponding to image size and the final dimension for the number of colors.  The -1 in this case indicates the dimension that will be automatically modified to keep the size of the new tensor the same as the original.
End of explanation
W_conv2 = weight_variable([5,5,32,64])
b_conv2 = bias_variable([64])
h_conv2 = tf.nn.relu(conv2d(h_pool1,W_conv2)+b_conv2)
h_pool2 = max_pool_2x2(h_conv2)
Explanation: Second Convolution Layer
We create a similar structure except now we have 32 inputs and 64 feature outputs for each 5x5 patch.
End of explanation
W_fc1 = weight_variable([7*7*64,1024])
b_fc1 = bias_variable([1024])
h_pool2_flat = tf.reshape(h_pool2, [-1,7*7*64])
h_fc1 = tf.nn.relu(tf.matmul(h_pool2_flat,W_fc1)+b_fc1)
Explanation: Densely Connected Layer
We have now done 2 2x2 convolutions which have reduced our image size to 7x7 since every 2x2 convolution effectively produces a new image that is half the size of the input image.
But for each 7x7 image, we now have 64 features.  So we will add a layer with 1024 neurons to allow processing on the entire image.
End of explanation
keep_prob = tf.placeholder(tf.float32)
h_fc1_drop = tf.nn.dropout(h_fc1,keep_prob)
Explanation: Dropout Layer
The dropout layer helps to reduce overfitting by dropping connections between neurons in the densely connected layers.  This paper has a nice discussion on the matter.
End of explanation
W_fc2 = weight_variable([1024, 10])
b_fc2 = bias_variable([10])
y_conv = tf.matmul(h_fc1_drop, W_fc2) + b_fc2
Explanation: Readout Layer
We add a layer that takes the output of our fully connected layer and does a softmax regression into our classes.
End of explanation
cross_entropy = tf.reduce_mean(
    tf.nn.softmax_cross_entropy_with_logits(labels=y_, logits=y_conv))
train_step = tf.train.AdamOptimizer(1e-4).minimize(cross_entropy)
correct_prediction = tf.equal(tf.argmax(y_conv,1), tf.argmax(y_,1))
accuracy = tf.reduce_mean(tf.cast(correct_prediction, tf.float32))
sess.run(tf.global_variables_initializer())
for i in range(10000):
  batch = mnist.train.next_batch(50)
  if i%100 == 0:
    train_accuracy = accuracy.eval(feed_dict={
        x:batch[0], y_: batch[1], keep_prob: 1.0})
    print("step %d, training accuracy %g"%(i, train_accuracy))
  train_step.run(feed_dict={x: batch[0], y_: batch[1], keep_prob: 0.5})
print("test accuracy %g"%accuracy.eval(feed_dict={
    x: mnist.test.images, y_: mnist.test.labels, keep_prob: 1.0}))
Explanation: Training
It should not that depending on the CPU available this could take some time to complete.
End of explanation
<END_TASK>