<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 Preprocessing
We will train with full data using multiple workers in cloud. To preprocess 8000 intances, it takes about 2~3 hours with 10 workers.
Step1:  Training
If it is the first time training for a while, warm-up takes about 5~10 minutes so the whole training takes about 20 minutes. On subsequent trainings, warm up time is about 1 minute.
Step2:  Check your job status. You can run
Step3:  Evaluation
First, we can check the TF summary events from training.
Step4:  Model Deployment
Deploy the model so we can do online prediction.
Step5:  Online Prediction
Step6:  Batch Prediction
Step7:  Check evaluation results. It is much better than local run results --- of course, we used much more data, and we ran more training steps.
Step8:  Clean up
<ASSISTANT_TASK:>
Python Code:
import mltoolbox.image.classification as model
from google.datalab.ml import *
import os
bucket = 'gs://' + datalab_project_id() + '-coast'
preprocessed_dir = bucket + '/preprocessed'
staging_dir = bucket + '/staging'
model_dir = bucket + '/model'
train_set = BigQueryDataSet('SELECT image_url, label FROM coast.train')
job = model.preprocess_async(train_set, preprocessed_dir, cloud={'num_workers': 5})
job.wait()
Explanation: Preprocessing
We will train with full data using multiple workers in cloud. To preprocess 8000 intances, it takes about 2~3 hours with 10 workers.
End of explanation
model.train_async(preprocessed_dir, 64, 8000, model_dir,
                  cloud=CloudTrainingConfig('us-central1', 'BASIC')).wait()
Explanation: Training
If it is the first time training for a while, warm-up takes about 5~10 minutes so the whole training takes about 20 minutes. On subsequent trainings, warm up time is about 1 minute.
End of explanation
tb_id = TensorBoard.start(model_dir)
Explanation: Check your job status. You can run:
Job('inception_train_170216_235400').describe()
to see job status.
End of explanation
summary = Summary(model_dir)
summary.list_events()
summary.plot('accuracy')
summary.plot('loss')
Explanation: Evaluation
First, we can check the TF summary events from training.
End of explanation
Models().create('coast')
ModelVersions('coast').deploy('v1', model_dir)
ModelVersions('coast').describe('v1')
Explanation: Model Deployment
Deploy the model so we can do online prediction.
End of explanation
# gs://tamucc_coastline/esi_images/IMG_2849_SecDE_Spr12.jpg,3B
# gs://tamucc_coastline/esi_images/IMG_0047_SecBC_Spr12.jpg,10A
# gs://tamucc_coastline/esi_images/IMG_0617_SecBC_Spr12.jpg,7
# gs://tamucc_coastline/esi_images/IMG_2034_SecEGH_Sum12_Pt2.jpg,10A
images = [
  'gs://tamucc_coastline/esi_images/IMG_2849_SecDE_Spr12.jpg',
  'gs://tamucc_coastline/esi_images/IMG_0047_SecBC_Spr12.jpg',
  'gs://tamucc_coastline/esi_images/IMG_0617_SecBC_Spr12.jpg',
  'gs://tamucc_coastline/esi_images/IMG_2034_SecEGH_Sum12_Pt2.jpg'
]
# resize=True because otherwise the images are too large to send for online prediction.
model.predict('coast.v1', images, resize=True, cloud=True)
Explanation: Online Prediction
End of explanation
eval_set = BigQueryDataSet(sql='select * from coast.eval')
model.batch_predict_async(eval_set, model_dir, output_bq_table='coast.eval_results_full',
                          cloud={'num_workers': 3, 'temp_location': staging_dir}).wait()
Explanation: Batch Prediction
End of explanation
%%bq query --name accuracy
SELECT
  target,
  SUM(CASE WHEN target=predicted THEN 1 ELSE 0 END) as correct,
  COUNT(*) as total,
  SUM(CASE WHEN target=predicted THEN 1 ELSE 0 END)/COUNT(*) as accuracy
FROM
  coast.eval_results_full
GROUP BY
  target
accuracy.execute().result()
%%bq query --name logloss
SELECT feature, AVG(-logloss) as logloss, count(*) as count FROM
(
  SELECT feature, CASE WHEN correct=1 THEN LOG(prob) ELSE LOG(1-prob) END as logloss
  FROM
  (
    SELECT
    target as feature, 
    CASE WHEN target=predicted THEN 1 ELSE 0 END as correct,
    target_prob as prob
    FROM coast.eval_results_full
  )
)
GROUP BY feature
FeatureSliceView().plot(logloss)
Explanation: Check evaluation results. It is much better than local run results --- of course, we used much more data, and we ran more training steps.
End of explanation
import shutil
import google.datalab.bigquery as bq
TensorBoard.stop(tb_id)
bq.Table('coast.eval_results_full').delete()
ModelVersions('coast').delete('v1')
Models().delete('coast')
!gsutil -m rm -r {preprocessed_dir}
!gsutil -m rm -r {model_dir}
Explanation: Clean up
End of explanation
<END_TASK>