<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 <h1> Structured data prediction using Cloud ML Engine with scikit-learn </h1>
This notebook illustrates
Step1:  Exploring dataset
Please see this notebook for more context on this problem and how the features were chosen.
Step3:  <h2> Creating a ML dataset using BigQuery </h2>
We can use BigQuery to create the training and evaluation datasets. Because of the masking (ultrasound vs. no ultrasound), the query itself is a little complex.
Step4:  <h2> Creating a scikit-learn model using random forests </h2>
Let's train the model locally
Step5:  Packaging up as a Python package
Note the %writefile in the cells above. I uncommented those and ran the cells to write out a model.py
The following cell writes out a task.py
Step6:  Try out the package on a subset of the data.
Step7:  <h2> Training on Cloud ML Engine </h2>
Submit the code to the ML Engine service
Step8:  The training finished in 20 minutes with a RMSE of 1.05 lbs.
<h2> Deploying the trained model </h2>
<p>
Deploying the trained model to act as a REST web service is a simple gcloud call.
Step9:  <h2> Using the model to predict </h2>
<p>
Send a JSON request to the endpoint of the service to make it predict a baby's weight ... Note that we need to send in an array of numbers in the same order as when we trained the model. You can sort of save some preprocessing by using sklearn's Pipeline, but we did our preprocessing with Pandas, so that is not an option.
<p>
So, let's find the order of columns
Step10:  As long as you send in the data in that order, it will work
Step11:  Hyperparameter tuning
Let's do a bunch of parallel trials to find good maxDepth and numTrees
Step12:  If you go to the GCP console and click on the job, you will see the trial information start to populating, with the lowest rmse trial listed first.  I got the best performance with these settings
<ASSISTANT_TASK:>
Python Code:
# change these to try this notebook out
BUCKET = 'cloud-training-demos-ml'
PROJECT = 'cloud-training-demos'
PROJECTNUMBER = '663413318684'
REGION = 'us-central1'
import os
os.environ['BUCKET'] = BUCKET
os.environ['PROJECT'] = PROJECT
os.environ['PROJECTNUMBER'] = PROJECTNUMBER
os.environ['REGION'] = REGION
%bash
gcloud config set project $PROJECT
gcloud config set compute/region $REGION
%%bash
if ! gsutil ls | grep -q gs://${BUCKET}/; then
  gsutil mb -l ${REGION} gs://${BUCKET}
fi
%bash
# Pandas will use this privatekey to access BigQuery on our behalf.
# Do NOT check in the private key into git!!!
# if you get a JWT grant error when using this key, create the key via gcp web console in IAM > Service Accounts section
KEYFILE=babyweight/trainer/privatekey.json
if [ ! -f $KEYFILE ]; then
  gcloud iam service-accounts keys create \
      --iam-account ${PROJECTNUMBER}-compute@developer.gserviceaccount.com \
      $KEYFILE
fi
KEYDIR='babyweight/trainer'
Explanation: <h1> Structured data prediction using Cloud ML Engine with scikit-learn </h1>
This notebook illustrates:
<ol>
<li> Creating datasets for Machine Learning using BigQuery
<li> Creating a model using scitkit learn 
<li> Training on Cloud ML Engine
<li> Deploying model
<li> Predicting with model
<li> Hyperparameter tuning of scikit-learn models
</ol>
Please see this notebook for more context on this problem and how the features were chosen.
End of explanation
#%writefile babyweight/trainer/model.py
# Copyright 2018 Google Inc. All Rights Reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
#     http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
Explanation: Exploring dataset
Please see this notebook for more context on this problem and how the features were chosen.
End of explanation
#%writefile -a babyweight/trainer/model.py
def create_queries():
  query_all = 
  WITH with_ultrasound AS (
    SELECT
      weight_pounds AS label,
      CAST(is_male AS STRING) AS is_male,
      mother_age,
      CAST(plurality AS STRING) AS plurality,
      gestation_weeks,
      FARM_FINGERPRINT(CONCAT(CAST(YEAR AS STRING), CAST(month AS STRING))) AS hashmonth
    FROM
      publicdata.samples.natality
    WHERE
      year > 2000
      AND gestation_weeks > 0
      AND mother_age > 0
      AND plurality > 0
      AND weight_pounds > 0
  ),
  without_ultrasound AS (
    SELECT
      weight_pounds AS label,
      'Unknown' AS is_male,
      mother_age,
      IF(plurality > 1, 'Multiple', 'Single') AS plurality,
      gestation_weeks,
      FARM_FINGERPRINT(CONCAT(CAST(YEAR AS STRING), CAST(month AS STRING))) AS hashmonth
    FROM
      publicdata.samples.natality
    WHERE
      year > 2000
      AND gestation_weeks > 0
      AND mother_age > 0
      AND plurality > 0
      AND weight_pounds > 0
  ),
  preprocessed AS (
    SELECT * from with_ultrasound
    UNION ALL
    SELECT * from without_ultrasound
  )
  SELECT
      label,
      is_male,
      mother_age,
      plurality,
      gestation_weeks
  FROM
      preprocessed
  
  train_query = "{} WHERE ABS(MOD(hashmonth, 4)) < 3".format(query_all)
  eval_query  = "{} WHERE ABS(MOD(hashmonth, 4)) = 3".format(query_all)
  return train_query, eval_query
print create_queries()[0]
#%writefile -a babyweight/trainer/model.py
def query_to_dataframe(query):
  import pandas as pd
  import pkgutil
  privatekey = pkgutil.get_data(KEYDIR, 'privatekey.json')
  print(privatekey[:200])
  return pd.read_gbq(query,
                     project_id=PROJECT,
                     dialect='standard',
                     private_key=privatekey)
def create_dataframes(frac):  
  # small dataset for testing
  if frac > 0 and frac < 1:
    sample = " AND RAND() < {}".format(frac)
  else:
    sample = ""
  train_query, eval_query = create_queries()
  train_query = "{} {}".format(train_query, sample)
  eval_query =  "{} {}".format(eval_query, sample)
  train_df = query_to_dataframe(train_query)
  eval_df = query_to_dataframe(eval_query)
  return train_df, eval_df
train_df, eval_df = create_dataframes(0.001)
train_df.describe()
eval_df.head()
Explanation: <h2> Creating a ML dataset using BigQuery </h2>
We can use BigQuery to create the training and evaluation datasets. Because of the masking (ultrasound vs. no ultrasound), the query itself is a little complex.
End of explanation
#%writefile -a babyweight/trainer/model.py
def input_fn(indf):
  import copy
  import pandas as pd
  df = copy.deepcopy(indf)
  # one-hot encode the categorical columns
  df["plurality"] = df["plurality"].astype(pd.api.types.CategoricalDtype(
                    categories=["Single","Multiple","1","2","3","4","5"]))
  df["is_male"] = df["is_male"].astype(pd.api.types.CategoricalDtype(
                  categories=["Unknown","false","true"]))
  # features, label
  label = df['label']
  del df['label']
  features = pd.get_dummies(df)
  return features, label
train_x, train_y = input_fn(train_df)
print(train_x[:5])
print(train_y[:5])
from sklearn.ensemble import RandomForestRegressor
estimator = RandomForestRegressor(max_depth=5, n_estimators=100, random_state=0)
estimator.fit(train_x, train_y)
import numpy as np
eval_x, eval_y = input_fn(eval_df)
eval_pred = estimator.predict(eval_x)
print(eval_pred[1000:1005])
print(eval_y[1000:1005])
print(np.sqrt(np.mean((eval_pred-eval_y)*(eval_pred-eval_y))))
#%writefile -a babyweight/trainer/model.py
def train_and_evaluate(frac, max_depth=5, n_estimators=100):
  import numpy as np
  # get data
  train_df, eval_df = create_dataframes(frac)
  train_x, train_y = input_fn(train_df)
  # train
  from sklearn.ensemble import RandomForestRegressor
  estimator = RandomForestRegressor(max_depth=max_depth, n_estimators=n_estimators, random_state=0)
  estimator.fit(train_x, train_y)
  # evaluate
  eval_x, eval_y = input_fn(eval_df)
  eval_pred = estimator.predict(eval_x)
  rmse = np.sqrt(np.mean((eval_pred-eval_y)*(eval_pred-eval_y)))
  print("Eval rmse={}".format(rmse))
  return estimator, rmse
#%writefile -a babyweight/trainer/model.py
def save_model(estimator, gcspath, name):
  from sklearn.externals import joblib
  import os, subprocess, datetime
  model = 'model.joblib'
  joblib.dump(estimator, model)
  model_path = os.path.join(gcspath, datetime.datetime.now().strftime(
    'export_%Y%m%d_%H%M%S'), model)
  subprocess.check_call(['gsutil', 'cp', model, model_path])
  return model_path
saved = save_model(estimator, 'gs://{}/babyweight/sklearn'.format(BUCKET), 'babyweight')
print saved
Explanation: <h2> Creating a scikit-learn model using random forests </h2>
Let's train the model locally
End of explanation
%writefile babyweight/trainer/task.py
# Copyright 2018 Google Inc. All Rights Reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
#     http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
import argparse
import os
import hypertune
import model
if __name__ == '__main__':
    parser = argparse.ArgumentParser()
    parser.add_argument(
        '--bucket',
        help = 'GCS path to output.',
        required = True
    )
    parser.add_argument(
        '--frac',
        help = 'Fraction of input to process',
        type = float,
        required = True
    )
    parser.add_argument(
        '--maxDepth',
        help = 'Depth of trees',
        type = int,
        default = 5
    )
    parser.add_argument(
        '--numTrees',
        help = 'Number of trees',
        type = int,
        default = 100
    )
    parser.add_argument(
        '--projectId',
        help = 'ID (not name) of your project',
        required = True
    )
    parser.add_argument(
        '--job-dir',
        help = 'output directory for model, automatically provided by gcloud',
        required = True
    )
    
    args = parser.parse_args()
    arguments = args.__dict__
    
    model.PROJECT = arguments['projectId']
    model.KEYDIR  = 'trainer'
    
    estimator, rmse = model.train_and_evaluate(arguments['frac'],
                                         arguments['maxDepth'],
                                         arguments['numTrees']
                                        )
    loc = model.save_model(estimator, 
                           arguments['job_dir'], 'babyweight')
    print("Saved model to {}".format(loc))
    
    # this is for hyperparameter tuning
    hpt = hypertune.HyperTune()
    hpt.report_hyperparameter_tuning_metric(
        hyperparameter_metric_tag='rmse',
        metric_value=rmse,
        global_step=0)
# done
!pip freeze | grep pandas
%writefile babyweight/setup.py
# Copyright 2018 Google Inc. All Rights Reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
#     http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
from setuptools import setup
setup(name='trainer',
      version='1.0',
      description='Natality, with sklearn',
      url='http://github.com/GoogleCloudPlatform/training-data-analyst',
      author='Google',
      author_email='nobody@google.com',
      license='Apache2',
      packages=['trainer'],
      ## WARNING! Do not upload this package to PyPI
      ## BECAUSE it contains a private key
      package_data={'': ['privatekey.json']},
      install_requires=[
          'pandas-gbq==0.3.0',
          'urllib3',
          'google-cloud-bigquery==0.29.0',
          'cloudml-hypertune'
      ],
      zip_safe=False)
Explanation: Packaging up as a Python package
Note the %writefile in the cells above. I uncommented those and ran the cells to write out a model.py
The following cell writes out a task.py
End of explanation
%bash
export PYTHONPATH=${PYTHONPATH}:${PWD}/babyweight
python -m trainer.task \
   --bucket=${BUCKET} --frac=0.001 --job-dir=gs://${BUCKET}/babyweight/sklearn --projectId $PROJECT
Explanation: Try out the package on a subset of the data.
End of explanation
%bash
RUNTIME_VERSION="1.8"
PYTHON_VERSION="2.7"
JOB_NAME=babyweight_skl_$(date +"%Y%m%d_%H%M%S")
JOB_DIR="gs://$BUCKET/babyweight/sklearn/${JOBNAME}"
gcloud ml-engine jobs submit training $JOB_NAME \
  --job-dir $JOB_DIR \
  --package-path $(pwd)/babyweight/trainer \
  --module-name trainer.task \
  --region us-central1 \
  --runtime-version=$RUNTIME_VERSION \
  --python-version=$PYTHON_VERSION \
  -- \
  --bucket=${BUCKET} --frac=0.1 --projectId $PROJECT
Explanation: <h2> Training on Cloud ML Engine </h2>
Submit the code to the ML Engine service
End of explanation
%bash
gsutil ls gs://${BUCKET}/babyweight/sklearn/ | tail -1
%bash
MODEL_NAME="babyweight"
MODEL_VERSION="skl"
MODEL_LOCATION=$(gsutil ls gs://${BUCKET}/babyweight/sklearn/ | tail -1)
echo "Deleting and deploying $MODEL_NAME $MODEL_VERSION from $MODEL_LOCATION ... this will take a few minutes"
#gcloud ml-engine versions delete ${MODEL_VERSION} --model ${MODEL_NAME}
#gcloud ml-engine models delete ${MODEL_NAME}
#gcloud ml-engine models create ${MODEL_NAME} --regions $REGION
gcloud alpha ml-engine versions create ${MODEL_VERSION} --model ${MODEL_NAME} --origin ${MODEL_LOCATION} \
    --framework SCIKIT_LEARN --runtime-version 1.8  --python-version=2.7
Explanation: The training finished in 20 minutes with a RMSE of 1.05 lbs.
<h2> Deploying the trained model </h2>
<p>
Deploying the trained model to act as a REST web service is a simple gcloud call.
End of explanation
data = []
for i in range(2):
  data.append([])
  for col in eval_x:
    # convert from numpy integers to standard integers
    data[i].append(int(np.uint64(eval_x[col][i]).item()))
print(eval_x.columns)
print(json.dumps(data))
Explanation: <h2> Using the model to predict </h2>
<p>
Send a JSON request to the endpoint of the service to make it predict a baby's weight ... Note that we need to send in an array of numbers in the same order as when we trained the model. You can sort of save some preprocessing by using sklearn's Pipeline, but we did our preprocessing with Pandas, so that is not an option.
<p>
So, let's find the order of columns:
End of explanation
from googleapiclient import discovery
from oauth2client.client import GoogleCredentials
import json
credentials = GoogleCredentials.get_application_default()
api = discovery.build('ml', 'v1', credentials=credentials)
request_data = {'instances':
  # [u'mother_age', u'gestation_weeks', u'is_male_Unknown', u'is_male_0',
  #     u'is_male_1', u'plurality_Single', u'plurality_Multiple',
  #     u'plurality_1', u'plurality_2', u'plurality_3', u'plurality_4',
  #     u'plurality_5']
  [[24, 38, 1, 0, 0, 1, 0, 0, 0, 0, 0, 0], 
   [34, 39, 0, 0, 1, 0, 0, 1, 0, 0, 0, 0]]
}
parent = 'projects/%s/models/%s/versions/%s' % (PROJECT, 'babyweight', 'skl')
response = api.projects().predict(body=request_data, name=parent).execute()
print "response={0}".format(response)
Explanation: As long as you send in the data in that order, it will work:
End of explanation
%writefile hyperparam.yaml
trainingInput:
  hyperparameters:
    goal: MINIMIZE
    maxTrials: 100
    maxParallelTrials: 5
    hyperparameterMetricTag: rmse
    params:
    - parameterName: maxDepth
      type: INTEGER
      minValue: 2
      maxValue: 8
      scaleType: UNIT_LINEAR_SCALE
    - parameterName: numTrees
      type: INTEGER
      minValue: 50
      maxValue: 150
      scaleType: UNIT_LINEAR_SCALE
%bash
RUNTIME_VERSION="1.8"
PYTHON_VERSION="2.7"
JOB_NAME=babyweight_skl_$(date +"%Y%m%d_%H%M%S")
JOB_DIR="gs://$BUCKET/babyweight/sklearn/${JOBNAME}"
gcloud ml-engine jobs submit training $JOB_NAME \
  --job-dir $JOB_DIR \
  --package-path $(pwd)/babyweight/trainer \
  --module-name trainer.task \
  --region us-central1 \
  --runtime-version=$RUNTIME_VERSION \
  --python-version=$PYTHON_VERSION \
  --config=hyperparam.yaml \
  -- \
  --bucket=${BUCKET} --frac=0.01 --projectId $PROJECT
Explanation: Hyperparameter tuning
Let's do a bunch of parallel trials to find good maxDepth and numTrees
End of explanation
%writefile largemachine.yaml
trainingInput:
  scaleTier: CUSTOM
  masterType: large_model
%bash
RUNTIME_VERSION="1.8"
PYTHON_VERSION="2.7"
JOB_NAME=babyweight_skl_$(date +"%Y%m%d_%H%M%S")
JOB_DIR="gs://$BUCKET/babyweight/sklearn/${JOBNAME}"
gcloud ml-engine jobs submit training $JOB_NAME \
  --job-dir $JOB_DIR \
  --package-path $(pwd)/babyweight/trainer \
  --module-name trainer.task \
  --region us-central1 \
  --runtime-version=$RUNTIME_VERSION \
  --python-version=$PYTHON_VERSION \
  --scale-tier=CUSTOM \
  --config=largemachine.yaml \
  -- \
  --bucket=${BUCKET} --frac=1 --projectId $PROJECT --maxDepth 8 --numTrees 90
Explanation: If you go to the GCP console and click on the job, you will see the trial information start to populating, with the lowest rmse trial listed first.  I got the best performance with these settings:
<pre>
      "hyperparameters": {
        "maxDepth": "8",
        "numTrees": "90"
      },
      "finalMetric": {
        "trainingStep": "1",
        "objectiveValue": 1.03123724461
      }
</pre>
Train on full dataset
Let's train on the full dataset with these hyperparameters.  I am using a larger machine (8 CPUS, 52 GB of memory).
End of explanation
<END_TASK>