<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 Creating a custom prediction routine with scikit-learn
<table align="left">
  <td>
    <a href="https
Step1:  Authenticate your GCP account
If you are using AI Platform Notebooks, your environment is already
authenticated. Skip this step.
If you are using Colab, run the cell below and follow the instructions
when prompted to authenticate your account via oAuth.
Otherwise, follow these steps
Step2:  Create a Cloud Storage bucket
The following steps are required, regardless of your notebook environment.
To deploy a custom prediction routine, you must upload your trained model
artifacts and your custom code to Cloud Storage.
Set the name of your Cloud Storage bucket below. It must be unique across all
Cloud Storage buckets. 
You may also change the REGION variable, which is used for operations
throughout the rest of this notebook. Make sure to choose a region where Cloud
AI Platform services are
available. You may
not use a Multi-Regional Storage bucket for training with AI Platform.
Step3:  Only if your bucket doesn't already exist
Step4:  Finally, validate access to your Cloud Storage bucket by examining its contents
Step5:  Building and training a scikit-learn model
Often, you can't use your data in its raw form to train a machine learning model. Even when you can, preprocessing the data before using it for training can sometimes improve your model.
Assuming that you expect the input for prediction to have the same format as your training data, you must apply identical preprocessing during training and prediction to ensure that your model makes consistent predictions.
In this section, create a preprocessing module and use it as part of training. Then export a preprocessor with characteristics learned during training to use later in your custom prediction routine.
Install dependencies for local training
Training locally (in the notebook) requires several dependencies
Step6:  Write your preprocessor
Scaling training data so each numerical feature column has a mean of 0 and a standard deviation of 1 can improve your model.
Create preprocess.py, which contains a class to do this scaling
Step7:  Notice that an instance of MySimpleScaler saves the means and standard deviations of each feature column on first use. Then it uses these summary statistics to scale data it encounters afterward.
This lets you store characteristics of the training distribution and use them for identical preprocessing at prediction time.
Train your model
Next, use preprocess.MySimpleScaler to preprocess the iris data, then train a model using scikit-learn.
At the end, export your trained model as a joblib (.joblib) file and export your MySimpleScaler instance as a pickle (.pkl) file
Step8:  Deploying a custom prediction routine
To deploy a custom prediction routine to serve predictions from your trained model, do the following
Step9:  Notice that, in addition to using the preprocessor that you defined during training, this predictor performs a postprocessing step that converts the prediction output from class indexes (0, 1, or 2) into label strings (the name of the flower type).
However, if the predictor receives a probabilities keyword argument with the value True, it returns a probability array instead, denoting the probability that each of the three classes is the correct label (according to the model). The last part of this tutorial shows how to provide a keyword argument during prediction.
Package your custom code
You must package predictor.py and preprocess.py as a .tar.gz source distribution package and provide the package to AI Platform so it can use your custom code to serve predictions.
Write the following setup.py to define your package
Step10:  Then run the following command to createdist/my_custom_code-0.1.tar.gz
Step11:  Upload model artifacts and custom code to Cloud Storage
Before you can deploy your model for serving, AI Platform needs access to the following files in Cloud Storage
Step12:  Deploy your custom prediction routine
Create a model resource and a version resource to deploy your custom prediction routine. First define variables with your resource names
Step13:  Then create your model
Step14:  Next, create a version. In this step, provide paths to the artifacts and custom code you uploaded to Cloud Storage
Step15:  Learn more about the options you must specify when you deploy a custom prediction routine.
Serving online predictions
Try out your deployment by sending an online prediction request. First, install the Google APIs Client Library for Python
Step16:  Then send two instances of iris data to your deployed version
Step17:  Note
Step18:  Cleaning up
To clean up all GCP resources used in this project, you can delete the GCP project you used for the tutorial.
Alternatively, you can clean up individual resources by running the following commands
<ASSISTANT_TASK:>
Python Code:
PROJECT_ID = "<your-project-id>" #@param {type:"string"}
! gcloud config set project $PROJECT_ID
Explanation: Creating a custom prediction routine with scikit-learn
<table align="left">
  <td>
    <a href="https://cloud.google.com/ml-engine/docs/scikit/custom-prediction-routine-scikit-learn">
      <img src="https://cloud.google.com/_static/images/cloud/icons/favicons/onecloud/super_cloud.png"
           alt="Google Cloud logo" width="32px"> Read on cloud.google.com
    </a>
  </td>
  <td>
    <a href="https://colab.research.google.com/github/GoogleCloudPlatform/cloudml-samples/blob/main/notebooks/scikit-learn/custom-prediction-routine-scikit-learn.ipynb">
      <img src="https://cloud.google.com/ml-engine/images/colab-logo-32px.png" alt="Colab logo"> Run in Colab
    </a>
  </td>
  <td>
    <a href="https://github.com/GoogleCloudPlatform/cloudml-samples/blob/main/notebooks/scikit-learn/custom-prediction-routine-scikit-learn.ipynb">
      <img src="https://cloud.google.com/ml-engine/images/github-logo-32px.png" alt="GitHub logo">
      View on GitHub
    </a>
  </td>
</table>
Beta
This is a beta release of custom prediction routines. This feature might be changed in backward-incompatible ways and is not subject to any SLA or deprecation policy.
Overview
This tutorial shows how to deploy a trained scikit-learn model to AI Platform and serve predictions using a custom prediction routine. This lets you customize how AI Platform responds to each prediction request.
In this example, you will use a custom prediction routine to preprocess
prediction input by scaling it, and to postprocess prediction output by converting class numbers to label strings.
The tutorial walks through several steps:
Training a simple scikit-learn model locally (in this notebook)
Creating and deploy a custom prediction routine to AI Platform
Serving prediction requests from that deployment
Dataset
This tutorial uses R.A. Fisher's Iris dataset, a small dataset that is popular for trying out machine learning techniques. Each instance has four numerical features, which are different measurements of a flower, and a target label that
marks it as one of three types of iris: Iris setosa, Iris versicolour, or Iris virginica.
This tutorial uses the copy of the Iris dataset included in the
scikit-learn library.
Objective
The goal is to train a model that uses a flower's measurements as input to predict what type of iris it is.
This tutorial focuses more on using this model with AI Platform than on
the design of the model itself.
Costs
This tutorial uses billable components of Google Cloud Platform (GCP):
AI Platform
Cloud Storage
Learn about AI Platform
pricing and Cloud Storage
pricing, and use the Pricing
Calculator
to generate a cost estimate based on your projected usage.
Before you begin
You must do several things before you can train and deploy a model in AI Platform:
Set up your local development environment.
Set up a GCP project with billing and the necessary
  APIs enabled.
Authenticate your GCP account in this notebook.
Create a Cloud Storage bucket to store your training package and your
  trained model.
Set up your local development environment
If you are using Colab or AI Platform Notebooks, your environment already
meets all the requirements to run this notebook. You can skip this step.
Otherwise, make sure your environment meets this notebook's requirements.
You need the following:
The Google Cloud SDK
Python 3
virtualenv
Jupyter notebook running in a virtual environment with Python 3
The Google Cloud guide to Setting up a Python development
environment and the Jupyter
installation guide provide detailed instructions
for meeting these requirements. The following steps provide a condensed set of
instructions:
Install and initialize the Cloud SDK.
Install Python 3.
Install
   virtualenv
   and create a virtual environment that uses Python 3.
Activate that environment and run pip install jupyter in a shell to install
   Jupyter.
Run jupyter notebook in a shell to launch Jupyter.
Open this notebook in the Jupyter Notebook Dashboard.
Set up your GCP project
The following steps are required, regardless of your notebook environment.
Select or create a GCP project.
Make sure that billing is enabled for your project.
Enable the AI Platform ("Cloud Machine Learning Engine") and Compute Engine
APIs.
Enter your project ID in the cell below. Then run the  cell to make sure the
Cloud SDK uses the right project for all the commands in this notebook.
Note: Jupyter runs lines prefixed with ! as shell commands, and it interpolates Python variables prefixed with $ into these commands.
End of explanation
import sys
# If you are running this notebook in Colab, run this cell and follow the
# instructions to authenticate your GCP account. This provides access to your
# Cloud Storage bucket and lets you submit training jobs and prediction
# requests.
if 'google.colab' in sys.modules:
  from google.colab import auth as google_auth
  google_auth.authenticate_user()
# If you are running this notebook locally, replace the string below with the
# path to your service account key and run this cell to authenticate your GCP
# account.
else:
  %env GOOGLE_APPLICATION_CREDENTIALS '<path-to-your-service-account-key.json>'
Explanation: Authenticate your GCP account
If you are using AI Platform Notebooks, your environment is already
authenticated. Skip this step.
If you are using Colab, run the cell below and follow the instructions
when prompted to authenticate your account via oAuth.
Otherwise, follow these steps:
In the GCP Console, go to the Create service account key
   page.
From the Service account drop-down list, select New service account.
In the Service account name field, enter a name.
From the Role drop-down list, select
   Machine Learning Engine > AI Platform Admin and
   Storage > Storage Object Admin.
Click Create. A JSON file that contains your key downloads to your
local environment.
Enter the path to your service account key as the
GOOGLE_APPLICATION_CREDENTIALS variable in the cell below and run the cell.
End of explanation
BUCKET_NAME = "<your-bucket-name>" #@param {type:"string"}
REGION = "us-central1" #@param {type:"string"}
Explanation: Create a Cloud Storage bucket
The following steps are required, regardless of your notebook environment.
To deploy a custom prediction routine, you must upload your trained model
artifacts and your custom code to Cloud Storage.
Set the name of your Cloud Storage bucket below. It must be unique across all
Cloud Storage buckets. 
You may also change the REGION variable, which is used for operations
throughout the rest of this notebook. Make sure to choose a region where Cloud
AI Platform services are
available. You may
not use a Multi-Regional Storage bucket for training with AI Platform.
End of explanation
! gsutil mb -l $REGION gs://$BUCKET_NAME
Explanation: Only if your bucket doesn't already exist: Run the following cell to create your Cloud Storage bucket.
End of explanation
! gsutil ls -al gs://$BUCKET_NAME
Explanation: Finally, validate access to your Cloud Storage bucket by examining its contents:
End of explanation
! pip install numpy>=1.16.0 scikit-learn==0.20.2
Explanation: Building and training a scikit-learn model
Often, you can't use your data in its raw form to train a machine learning model. Even when you can, preprocessing the data before using it for training can sometimes improve your model.
Assuming that you expect the input for prediction to have the same format as your training data, you must apply identical preprocessing during training and prediction to ensure that your model makes consistent predictions.
In this section, create a preprocessing module and use it as part of training. Then export a preprocessor with characteristics learned during training to use later in your custom prediction routine.
Install dependencies for local training
Training locally (in the notebook) requires several dependencies:
End of explanation
%%writefile preprocess.py
import numpy as np
class MySimpleScaler(object):
  def __init__(self):
    self._means = None
    self._stds = None
  def preprocess(self, data):
    if self._means is None: # during training only
      self._means = np.mean(data, axis=0)
    if self._stds is None: # during training only
      self._stds = np.std(data, axis=0)
      if not self._stds.all():
        raise ValueError('At least one column has standard deviation of 0.')
    return (data - self._means) / self._stds
Explanation: Write your preprocessor
Scaling training data so each numerical feature column has a mean of 0 and a standard deviation of 1 can improve your model.
Create preprocess.py, which contains a class to do this scaling:
End of explanation
import pickle
from sklearn.datasets import load_iris
from sklearn.ensemble import RandomForestClassifier
from sklearn.externals import joblib
from preprocess import MySimpleScaler
iris = load_iris()
scaler = MySimpleScaler()
X = scaler.preprocess(iris.data)
y = iris.target
model = RandomForestClassifier()
model.fit(X, y)
joblib.dump(model, 'model.joblib')
with open ('preprocessor.pkl', 'wb') as f:
  pickle.dump(scaler, f)
Explanation: Notice that an instance of MySimpleScaler saves the means and standard deviations of each feature column on first use. Then it uses these summary statistics to scale data it encounters afterward.
This lets you store characteristics of the training distribution and use them for identical preprocessing at prediction time.
Train your model
Next, use preprocess.MySimpleScaler to preprocess the iris data, then train a model using scikit-learn.
At the end, export your trained model as a joblib (.joblib) file and export your MySimpleScaler instance as a pickle (.pkl) file:
End of explanation
%%writefile predictor.py
import os
import pickle
import numpy as np
from sklearn.datasets import load_iris
from sklearn.externals import joblib
class MyPredictor(object):
  def __init__(self, model, preprocessor):
    self._model = model
    self._preprocessor = preprocessor
    self._class_names = load_iris().target_names
  def predict(self, instances, **kwargs):
    inputs = np.asarray(instances)
    preprocessed_inputs = self._preprocessor.preprocess(inputs)
    if kwargs.get('probabilities'):
      probabilities = self._model.predict_proba(preprocessed_inputs)
      return probabilities.tolist()
    else:
      outputs = self._model.predict(preprocessed_inputs)
      return [self._class_names[class_num] for class_num in outputs]
  @classmethod
  def from_path(cls, model_dir):
    model_path = os.path.join(model_dir, 'model.joblib')
    model = joblib.load(model_path)
    preprocessor_path = os.path.join(model_dir, 'preprocessor.pkl')
    with open(preprocessor_path, 'rb') as f:
      preprocessor = pickle.load(f)
    return cls(model, preprocessor)
Explanation: Deploying a custom prediction routine
To deploy a custom prediction routine to serve predictions from your trained model, do the following:
Create a custom predictor to handle requests
Package your predictor and your preprocessing module
Upload your model artifacts and your custom code to Cloud Storage
Deploy your custom prediction routine to AI Platform
Create a custom predictor
To deploy a custom prediction routine, you must create a class that implements
the Predictor interface. This tells AI Platform how to load your model and how to handle prediction requests.
Write the following code to predictor.py:
End of explanation
%%writefile setup.py
from setuptools import setup
setup(
    name='my_custom_code',
    version='0.1',
    scripts=['predictor.py', 'preprocess.py'])
Explanation: Notice that, in addition to using the preprocessor that you defined during training, this predictor performs a postprocessing step that converts the prediction output from class indexes (0, 1, or 2) into label strings (the name of the flower type).
However, if the predictor receives a probabilities keyword argument with the value True, it returns a probability array instead, denoting the probability that each of the three classes is the correct label (according to the model). The last part of this tutorial shows how to provide a keyword argument during prediction.
Package your custom code
You must package predictor.py and preprocess.py as a .tar.gz source distribution package and provide the package to AI Platform so it can use your custom code to serve predictions.
Write the following setup.py to define your package:
End of explanation
! python setup.py sdist --formats=gztar
Explanation: Then run the following command to createdist/my_custom_code-0.1.tar.gz:
End of explanation
! gsutil cp ./dist/my_custom_code-0.1.tar.gz gs://$BUCKET_NAME/custom_prediction_routine_tutorial/my_custom_code-0.1.tar.gz
! gsutil cp model.joblib preprocessor.pkl gs://$BUCKET_NAME/custom_prediction_routine_tutorial/model/
Explanation: Upload model artifacts and custom code to Cloud Storage
Before you can deploy your model for serving, AI Platform needs access to the following files in Cloud Storage:
model.joblib (model artifact)
preprocessor.pkl (model artifact)
my_custom_code-0.1.tar.gz (custom code)
Model artifacts must be stored together in a model directory, which your
Predictor can access as the model_dir argument in its from_path class
method. The custom
code does not need to be in the same directory. Run the following commands to
upload your files:
End of explanation
MODEL_NAME = 'IrisPredictor'
VERSION_NAME = 'v1'
Explanation: Deploy your custom prediction routine
Create a model resource and a version resource to deploy your custom prediction routine. First define variables with your resource names:
End of explanation
! gcloud ai-platform models create $MODEL_NAME \
  --regions $REGION
Explanation: Then create your model:
End of explanation
# --quiet automatically installs the beta component if it isn't already installed 
! gcloud --quiet beta ai-platform versions create $VERSION_NAME \
  --model $MODEL_NAME \
  --runtime-version 1.13 \
  --python-version 3.5 \
  --origin gs://$BUCKET_NAME/custom_prediction_routine_tutorial/model/ \
  --package-uris gs://$BUCKET_NAME/custom_prediction_routine_tutorial/my_custom_code-0.1.tar.gz \
  --prediction-class predictor.MyPredictor
Explanation: Next, create a version. In this step, provide paths to the artifacts and custom code you uploaded to Cloud Storage:
End of explanation
! pip install --upgrade google-api-python-client
Explanation: Learn more about the options you must specify when you deploy a custom prediction routine.
Serving online predictions
Try out your deployment by sending an online prediction request. First, install the Google APIs Client Library for Python:
End of explanation
import googleapiclient.discovery
instances = [
  [6.7, 3.1, 4.7, 1.5],
  [4.6, 3.1, 1.5, 0.2],
]
service = googleapiclient.discovery.build('ml', 'v1')
name = 'projects/{}/models/{}/versions/{}'.format(PROJECT_ID, MODEL_NAME, VERSION_NAME)
response = service.projects().predict(
    name=name,
    body={'instances': instances}
).execute()
if 'error' in response:
    raise RuntimeError(response['error'])
else:
  print(response['predictions'])
Explanation: Then send two instances of iris data to your deployed version:
End of explanation
response = service.projects().predict(
    name=name,
    body={'instances': instances, 'probabilities': True}
).execute()
if 'error' in response:
    raise RuntimeError(response['error'])
else:
  print(response['predictions'])
Explanation: Note: This code uses the credentials you set up during the authentication step to make the online prediction request.
Sending keyword arguments
When you send a prediction request to a custom prediction routine, you can provide additional fields on your request body. The Predictor's predict method receives these as fields of the **kwargs dictionary.
The following code sends the same request as before, but this time it adds a probabilities field to the request body:
End of explanation
# Delete version resource
! gcloud ai-platform versions delete $VERSION_NAME --quiet --model $MODEL_NAME 
# Delete model resource
! gcloud ai-platform models delete $MODEL_NAME --quiet
# Delete Cloud Storage objects that were created
! gsutil -m rm -r gs://$BUCKET_NAME/custom_prediction_routine_tutorial
Explanation: Cleaning up
To clean up all GCP resources used in this project, you can delete the GCP project you used for the tutorial.
Alternatively, you can clean up individual resources by running the following commands:
End of explanation
<END_TASK>