<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 Order of magnitude faster training for image classification
Step1:  Preprocess
Preprocessing uses a Dataflow pipeline to convert the image format, resize images, and run the converted image through a pre-trained model to get the features or embeddings. You can also do this step using alternate technologies like Spark or plain Python code if you like. 
The %%ml preprocess command simplifies this task. Check out the parameters shown using --usage flag first and then run the command.
If you hit "PERMISSION_DENIED" when running the following cell, you need to enable Cloud DataFlow API (url is shown in error message). 
The DataFlow job usually takes about 20 min to complete.
Step2:  Train
Note that the command remains the same as that in the "local" version.
Step3:  Check your job status by running (replace the job id from the one shown above)
Step4:  Predict
Deploy the model and run online predictions. The deployment takes about 2 ~ 5 minutes.
Step5:  Online prediction is currently in alpha, it helps to ensure a warm start if the first call fails.
Step6:  Batch Predict
Step7:  Clean up
<ASSISTANT_TASK:>
Python Code:
import mltoolbox.image.classification as model
from google.datalab.ml import *
bucket = 'gs://' + datalab_project_id() + '-lab'
preprocess_dir = bucket + '/flowerpreprocessedcloud'
model_dir = bucket + '/flowermodelcloud'
staging_dir = bucket + '/staging'
!gsutil mb $bucket
Explanation: Order of magnitude faster training for image classification: Part II
Transfer learning using Inception Package - Cloud Run Experience
This notebook continues the codifies the capabilities discussed in this blog post. In a nutshell, it uses the pre-trained inception model as a starting point and then uses transfer learning to train it further on additional, customer-specific images. For explanation, simple flower images are used. Compared to training from scratch, the time and costs are drastically reduced.
This notebook does preprocessing, training and prediction by calling CloudML API instead of running them in the Datalab container.  The purpose of local work is to do some initial prototyping and debugging on small scale data - often by taking a suitable (say 0.1 - 1%) sample of the full data. The same basic steps can then be repeated with much larger datasets in cloud.
Setup
First run the following steps only if you are running Datalab from your local desktop or laptop (not running Datalab from a GCE VM):
Make sure you have a GCP project which is enabled for Machine Learning API and Dataflow API.
Run "%datalab project set --project [project-id]" to set the default project in Datalab.
If you run Datalab from a GCE VM, then make sure the project of the GCE VM is enabled for Machine Learning API and Dataflow API.
End of explanation
train_set = CsvDataSet('gs://cloud-datalab/sampledata/flower/train1000.csv', schema='image_url:STRING,label:STRING')
preprocess_job = model.preprocess_async(train_set, preprocess_dir, cloud={'num_workers': 10})
preprocess_job.wait() # Alternatively, you can query the job status by train_job.state. The wait() call blocks the notebook execution.
Explanation: Preprocess
Preprocessing uses a Dataflow pipeline to convert the image format, resize images, and run the converted image through a pre-trained model to get the features or embeddings. You can also do this step using alternate technologies like Spark or plain Python code if you like. 
The %%ml preprocess command simplifies this task. Check out the parameters shown using --usage flag first and then run the command.
If you hit "PERMISSION_DENIED" when running the following cell, you need to enable Cloud DataFlow API (url is shown in error message). 
The DataFlow job usually takes about 20 min to complete.
End of explanation
train_job = model.train_async(preprocess_dir, 30, 1000, model_dir, cloud=CloudTrainingConfig('us-central1', 'BASIC'))
train_job.wait() # Alternatively, you can query the job status by train_job.state. The wait() call blocks the notebook execution.
Explanation: Train
Note that the command remains the same as that in the "local" version.
End of explanation
tb_id = TensorBoard.start(model_dir)
Explanation: Check your job status by running (replace the job id from the one shown above):
Job('image_classification_train_170307_002934').describe()
Tensorboard works too with GCS path. Note that the data will show up usually a minute after tensorboard starts with GCS path.
End of explanation
Models().create('flower')
ModelVersions('flower').deploy('beta1', model_dir)
Explanation: Predict
Deploy the model and run online predictions. The deployment takes about 2 ~ 5 minutes.
End of explanation
images = [
    'gs://cloud-ml-data/img/flower_photos/daisy/15207766_fc2f1d692c_n.jpg',
    'gs://cloud-ml-data/img/flower_photos/tulips/6876631336_54bf150990.jpg'
]
# set resize=True to avoid sending large data in prediction request.
model.predict('flower.beta1', images, resize=True, cloud=True)
Explanation: Online prediction is currently in alpha, it helps to ensure a warm start if the first call fails.
End of explanation
import google.datalab.bigquery as bq
bq.Dataset('flower').create()
eval_set = CsvDataSet('gs://cloud-datalab/sampledata/flower/eval670.csv', schema='image_url:STRING,label:STRING')
batch_predict_job = model.batch_predict_async(eval_set, model_dir, output_bq_table='flower.eval_results_full',
                                              cloud={'temp_location': staging_dir})
batch_predict_job.wait()
%%bq query --name wrong_prediction
SELECT * FROM flower.eval_results_full WHERE target != predicted
wrong_prediction.execute().result()
ConfusionMatrix.from_bigquery('flower.eval_results_full').plot()
%%bq query --name accuracy
SELECT
  target,
  SUM(CASE WHEN target=predicted THEN 1 ELSE 0 END) as correct,
  COUNT(*) as total,
  SUM(CASE WHEN target=predicted THEN 1 ELSE 0 END)/COUNT(*) as accuracy
FROM
  flower.eval_results_full
GROUP BY
  target
accuracy.execute().result()
%%bq query --name logloss
SELECT feature, AVG(-logloss) as logloss, count(*) as count FROM
(
SELECT feature, CASE WHEN correct=1 THEN LOG(prob) ELSE LOG(1-prob) END as logloss
FROM
(
SELECT
target as feature, 
CASE WHEN target=predicted THEN 1 ELSE 0 END as correct,
target_prob as prob
FROM flower.eval_results_full))
GROUP BY feature
FeatureSliceView().plot(logloss)
Explanation: Batch Predict
End of explanation
ModelVersions('flower').delete('beta1')
Models().delete('flower')
!gsutil -m rm -r {preprocess_dir}
!gsutil -m rm -r {model_dir}
Explanation: Clean up
End of explanation
<END_TASK>