<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 Fashion MNIST Image Classification with TensorFlow on Cloud ML Engine
This notebook demonstrates how to train a deep neural network model for image classification and deploy it as an Application Programming Interface (API) (i.e. web service) for online predictions.
Step1:  Run as a Python module
Now since we want to run our code on Cloud ML Engine, we've packaged it as a Python module.
The model.py and task.py files containing the model code are in <a href="fashionmodel/trainer">fashionmodel/trainer</a>
Step2:  Training should finish in just a few seconds because it ran outside of the Jupyter's runtime, as an independent process of the node's operating system. You can explore the training metrics using TensorBoard.
Step3:  Next, let's use Cloud ML Engine so we can train on GPU
Step4:  Monitoring training with TensorBoard
Use this cell to launch tensorboard
Step5:  Deploying and predicting with model
Deploy the model
Step6:  The previous step of deploying the model can take a few minutes. If it is successful, you should see an output similar to this one
Step7:  To predict with the model, save one of the test images as a JavaScript Object Notation (JSON) file. Also, take a look at it as a graphic and notice the expected class value in the title.
Step8:  Here's how the same image looks when it saved in the test.json file for use with the prediction API.
Step9:  Send the file to the prediction service and check whether the model you trained returns the correct prediction.
<ASSISTANT_TASK:>
Python Code:
import os
PROJECT = 'my-project-id' # REPLACE WITH YOUR PROJECT ID
BUCKET = 'my-bucket-name' # REPLACE WITH YOUR BUCKET NAME
REGION = 'us-central1' # REPLACE WITH YOUR BUCKET REGION e.g. us-central1
MODEL_TYPE='dnn'  #  'dnn' or 'cnn'
# do not change these
os.environ['PROJECT'] = PROJECT
os.environ['BUCKET'] = BUCKET
os.environ['REGION'] = REGION
os.environ['MODEL_TYPE'] = MODEL_TYPE
os.environ['TFVERSION'] = '1.8'  # Tensorflow version
%bash
gcloud config set project $PROJECT
gcloud config set compute/region $REGION
Explanation: Fashion MNIST Image Classification with TensorFlow on Cloud ML Engine
This notebook demonstrates how to train a deep neural network model for image classification and deploy it as an Application Programming Interface (API) (i.e. web service) for online predictions.
End of explanation
%bash
rm -rf fashionmodel.tar.gz fashion_trained
gcloud ml-engine local train \
   --module-name=trainer.task \
   --package-path=${PWD}/fashionmodel/trainer \
   -- \
   --output_dir=${PWD}/fashion_trained \
   --train_steps=1000 \
   --learning_rate=0.01 \
   --train_batch_size=512 \
   --model=$MODEL_TYPE
Explanation: Run as a Python module
Now since we want to run our code on Cloud ML Engine, we've packaged it as a Python module.
The model.py and task.py files containing the model code are in <a href="fashionmodel/trainer">fashionmodel/trainer</a>
End of explanation
from google.datalab.ml import TensorBoard
TensorBoard().start('fashion_trained'.format(BUCKET, MODEL_TYPE))
for pid in TensorBoard.list()['pid']:
  TensorBoard().stop(pid)
  print 'Stopped TensorBoard with pid {}'.format(pid)
Explanation: Training should finish in just a few seconds because it ran outside of the Jupyter's runtime, as an independent process of the node's operating system. You can explore the training metrics using TensorBoard.
End of explanation
%bash
OUTDIR=gs://${BUCKET}/fashion/trained_${MODEL_TYPE}
JOBNAME=fashion_${MODEL_TYPE}_$(date -u +%y%m%d_%H%M%S)
echo $OUTDIR $REGION $JOBNAME
gsutil -m rm -rf $OUTDIR
gcloud ml-engine jobs submit training $JOBNAME \
   --region=$REGION \
   --module-name=trainer.task \
   --package-path=${PWD}/fashionmodel/trainer \
   --job-dir=$OUTDIR \
   --staging-bucket=gs://$BUCKET \
   --scale-tier=BASIC_GPU \
   --runtime-version=$TFVERSION \
   -- \
   --output_dir=$OUTDIR \
   --train_steps=1000 --learning_rate=0.01 --train_batch_size=512 \
   --model=$MODEL_TYPE
Explanation: Next, let's use Cloud ML Engine so we can train on GPU: --scale-tier=BASIC_GPU and deploy the model as a web service API
Note that GPU speed up depends on the model type. You'll notice that more complex models train substantially faster on GPUs. When you are working with simple models that take just seconds to minutes to train on a single node, keep in mind that Cloud ML Engine introduces a few minutes of overhead for training job setup & teardown.
End of explanation
from google.datalab.ml import TensorBoard
TensorBoard().start('gs://{}/fashion/trained_{}'.format(BUCKET, MODEL_TYPE))
for pid in TensorBoard.list()['pid']:
  TensorBoard().stop(pid)
  print 'Stopped TensorBoard with pid {}'.format(pid)
Explanation: Monitoring training with TensorBoard
Use this cell to launch tensorboard
End of explanation
%bash
MODEL_NAME="fashion"
MODEL_VERSION=${MODEL_TYPE}
MODEL_LOCATION=$(gsutil ls gs://${BUCKET}/fashion/trained_${MODEL_TYPE}/export/exporter | tail -1)
echo "Deleting and deploying $MODEL_NAME $MODEL_VERSION from $MODEL_LOCATION ... this will take a few minutes"
#gcloud ml-engine versions delete ${MODEL_VERSION} --model ${MODEL_NAME}
#gcloud ml-engine models delete ${MODEL_NAME}
gcloud ml-engine models create ${MODEL_NAME} --regions $REGION
gcloud ml-engine versions create ${MODEL_VERSION} --model ${MODEL_NAME} --origin ${MODEL_LOCATION} --runtime-version=$TFVERSION
Explanation: Deploying and predicting with model
Deploy the model:
End of explanation
import tensorflow as tf
(train_images, train_labels), (test_images, test_labels) = tf.keras.datasets.fashion_mnist.load_data()
LABELS = ['T-shirt/top', 'Trouser', 'Pullover', 'Dress', 'Coat', 'Sandal', 'Shirt', 'Sneaker', 'Bag', 'Ankle boot']
Explanation: The previous step of deploying the model can take a few minutes. If it is successful, you should see an output similar to this one:
<pre>
Created ml engine model [projects/qwiklabs-gcp-27eb45524d98e9a5/models/fashion].
Creating version (this might take a few minutes)......
...................................................................................................................done.
</pre>
Next, download a local copy of the Fashion MNIST dataset to use with Cloud ML Engine for predictions.
End of explanation
HEIGHT=28
WIDTH=28
IMGNO=12 #CHANGE THIS to get different images
#Convert raw image data to a test.json file and persist it to disk
import json, codecs
jsondata = {'image': test_images[IMGNO].reshape(HEIGHT, WIDTH).tolist()}
json.dump(jsondata, codecs.open('test.json', 'w', encoding='utf-8'))
#Take a look at a sample image and the correct label from the test dataset
import matplotlib.pyplot as plt
plt.imshow(test_images[IMGNO].reshape(HEIGHT, WIDTH))
title = plt.title('{} / Class #{}'.format(LABELS[test_labels[IMGNO]], test_labels[IMGNO]))
Explanation: To predict with the model, save one of the test images as a JavaScript Object Notation (JSON) file. Also, take a look at it as a graphic and notice the expected class value in the title.
End of explanation
%bash
cat test.json
Explanation: Here's how the same image looks when it saved in the test.json file for use with the prediction API.
End of explanation
%bash
gcloud ml-engine predict \
   --model=fashion \
   --version=${MODEL_TYPE} \
   --json-instances=./test.json
Explanation: Send the file to the prediction service and check whether the model you trained returns the correct prediction.
End of explanation
<END_TASK>