<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 <table align="left">
    <td>
        <a href="https
Step1:  Set up your Google Cloud project
The following steps are required, regardless of your notebook environment.
Select or create a Google Cloud project. When you first create an account, you get a $300 free credit towards your compute/storage costs.
Make sure that billing is enabled for your project.
Enable the AI Platform APIs
If running locally on your own machine, you will need to install the Google Cloud SDK.
Note
Step2:  Install CloudTuner
Download and install CloudTuner from tensorflow-enterprise-addons wheel package.
Step3:  Restart the Kernel
We will automatically restart your kernel so the notebook has access to the packages you installed.
Step4:  Import libraries and define constants
Step5:  Tutorial
Prepare Data
For this tutorial, we will use a subset (10000 examples) from the MNIST dataset.
Step6:  Define model building function
Next, we will define the hyperparameter model building function like one does for KerasTuner, where the following are tunable
Step7:  Instantiate CloudTuner
Next, we instantiate an instance of the CloudTuner. We will define our tuning hyperparameters and pass them into the constructor as the parameter hyperparameters.
We also set the objective ('accuracy') to measure the performance of each trial, and we shall keep the number of trials small (5) for the purpose of this demonstration.
Step8:  Let's use the search_space_summary() method to display what the search space for this optimization study looks like.
Step9:  Search
Let's now execute our search for this optimization study with the search() method. This method takes the same parameters as the fit() method in TF.keras API model instance.
Step10:  Results
Next, we use the results_summary() method to get a summary of the trials that were tried in this optimization study.
Step11:  Get the Best Model
Now, let's get the best model from the study using the get_best_models() method. The parameter num specifies the topmost number of models. In our case, we set it to 1 for the best overall model. The method returns a list (of models), so we use index of 0 to get the model out of the list.
Step12:  Tutorial
Step13:  Load MNIST Data
Step15:  Build training pipeline
Build a training and evaluation pipeline using ds.map, ds.cache, ds.shuffle, ds.batch, and ds.prefetch. For more details on building high performance pipelines refer to data performance
Step16:  Create and train the model
Step17:  Tutorial
Step18:  Instantiate CloudTuner
Next, we instantiate an instance of the CloudTuner. In this instantiation, we replace the hyperparameters and objective parameters with the study_config parameter.
Step19:  Let's use the search_space_summary() method to display what the search space for this optimization study looks like.
Step20:  Search
Let's now execute our search for this optimization study with the search() method.
Step21:  Results
Now let's use the results_summary() method to get a summary of the trials that were tried in this optimization study.
Step23:  Tutorial
Step24:  Search
Let's now execute multiple search loops in parallel for this study with the search() method.
Step25:  Results
Now let's use the results_summary() method to get a summary of the trials (from all the search loops) that were tried in this optimization study.
<ASSISTANT_TASK:>
Python Code:
! pip install google-cloud
! pip install google-cloud-storage
! pip install requests
! pip install tensorflow_datasets
Explanation: <table align="left">
    <td>
        <a href="https://colab.research.google.com/github/GoogleCloudPlatform/ai-platform-samples/blob/master/notebooks/samples/optimizer/ai_platform_optimizer_tuner.ipynb">
            <img src="https://cloud.google.com/ml-engine/images/colab-logo-32px.png" alt="Colab logo"> Run in Colab
        </a>
    </td>
    <td>
        <a href="https://github.com/GoogleCloudPlatform/ai-platform-samples/blob/master/notebooks/samples/optimizer/ai_platform_optimizer_tuner.ipynb">
            <img src="https://cloud.google.com/ml-engine/images/github-logo-32px.png" alt="GitHub logo">View on GitHub
        </a>
     </td>
</table>
Overview
This tutorial demonstrates AI Platform's CloudTuner service.
Objective
CloudTuner is implemented into the KerasTuner and uses CAIP Optimizer Beta (Vizier as a backend) as an oracle to get suggested trials, run trials, etc. The usage of CloudTuner is the same as KerasTuner and additionally accept Optimizer's study_config as an alternative input.
Costs
This tutorial uses billable components of Google Cloud:
AI Platform Training
Cloud Storage
Learn about AI Platform Training
pricing and Cloud Storage
pricing, and use the Pricing
Calculator
to generate a cost estimate based on your projected usage.
PIP install packages and dependencies
Install additional dependencies not installed in the notebook environment.
Use the latest major GA version of the framework.
End of explanation
import sys
# If you are running this notebook in Colab, run this cell and follow the
# instructions to authenticate your Google Cloud account. This provides access
# to your Cloud Storage bucket and lets you submit training jobs and prediction
# requests.
if 'google.colab' in sys.modules:
    from google.colab import auth as google_auth
    google_auth.authenticate_user()
# If you are running this tutorial in a notebook locally, replace the string
# below with the path to your service account key and run this cell to
# authenticate your Google Cloud account.
else:
    %env GOOGLE_APPLICATION_CREDENTIALS your_path_to_credentials.json
# Log in to your account on Google Cloud
! gcloud auth application-default login
! gcloud auth login
Explanation: Set up your Google Cloud project
The following steps are required, regardless of your notebook environment.
Select or create a Google Cloud project. When you first create an account, you get a $300 free credit towards your compute/storage costs.
Make sure that billing is enabled for your project.
Enable the AI Platform APIs
If running locally on your own machine, you will need to install the Google Cloud SDK.
Note: Jupyter runs lines prefixed with ! as shell commands, and it interpolates Python variables prefixed with $ into these commands.
Authenticate your Google Cloud account
If you are using AI Platform Notebooks, your environment is already
authenticated. Skip these steps.
End of explanation
! pip install tensorflow-enterprise-addons
Explanation: Install CloudTuner
Download and install CloudTuner from tensorflow-enterprise-addons wheel package.
End of explanation
# Restart the kernel after pip installs
import IPython
app = IPython.Application.instance()
app.kernel.do_shutdown(True)
Explanation: Restart the Kernel
We will automatically restart your kernel so the notebook has access to the packages you installed.
End of explanation
from tensorflow_enterprise_addons import cloudtuner
import kerastuner
REGION = 'us-central1'
PROJECT_ID = '[your-project-id]' #@param {type:"string"}
! gcloud config set project $PROJECT_ID
Explanation: Import libraries and define constants
End of explanation
from tensorflow.keras.datasets import mnist
(x, y), (val_x, val_y) = mnist.load_data()
x = x.astype('float32') / 255.
val_x = val_x.astype('float32') / 255.
x = x[:10000]
y = y[:10000]
Explanation: Tutorial
Prepare Data
For this tutorial, we will use a subset (10000 examples) from the MNIST dataset.
End of explanation
from tensorflow.keras import Sequential
from tensorflow.keras.layers import Flatten, Dense
from tensorflow.keras.optimizers import Adam
def build_model(hp):
    model = Sequential()
    model.add(Flatten(input_shape=(28, 28)))
    # the number of layers is tunable
    for _ in range(hp.get('num_layers')):
        model.add(Dense(units=64, activation='relu'))
    model.add(Dense(10, activation='softmax'))
    # the learning rate is tunable
    model.compile(
        optimizer=Adam(lr=hp.get('learning_rate')),
        loss='sparse_categorical_crossentropy',
        metrics=['accuracy'])
    return model
Explanation: Define model building function
Next, we will define the hyperparameter model building function like one does for KerasTuner, where the following are tunable:
- number of layers
- the learning rate
Note that CloudTuner does not support adding hyperparameters in the model building function. Instead, the search space is configured by passing a hyperparameters argument when instantiating (constructing) the tuner.
End of explanation
# Configure the search space
HPS = kerastuner.engine.hyperparameters.HyperParameters()
HPS.Float('learning_rate', min_value=1e-4, max_value=1e-2, sampling='log')
HPS.Int('num_layers', 2, 10)
tuner = cloudtuner.CloudTuner(
    build_model,
    project_id=PROJECT_ID,
    region=REGION,
    objective='accuracy',
    hyperparameters=HPS,
    max_trials=5,
    directory='tmp_dir/1')
Explanation: Instantiate CloudTuner
Next, we instantiate an instance of the CloudTuner. We will define our tuning hyperparameters and pass them into the constructor as the parameter hyperparameters.
We also set the objective ('accuracy') to measure the performance of each trial, and we shall keep the number of trials small (5) for the purpose of this demonstration.
End of explanation
tuner.search_space_summary()
Explanation: Let's use the search_space_summary() method to display what the search space for this optimization study looks like.
End of explanation
tuner.search(x=x, y=y, epochs=10, validation_data=(val_x, val_y))
Explanation: Search
Let's now execute our search for this optimization study with the search() method. This method takes the same parameters as the fit() method in TF.keras API model instance.
End of explanation
tuner.results_summary()
Explanation: Results
Next, we use the results_summary() method to get a summary of the trials that were tried in this optimization study.
End of explanation
model = tuner.get_best_models(num_models=1)[0]
print(model)
print(model.weights)
Explanation: Get the Best Model
Now, let's get the best model from the study using the get_best_models() method. The parameter num specifies the topmost number of models. In our case, we set it to 1 for the best overall model. The method returns a list (of models), so we use index of 0 to get the model out of the list.
End of explanation
import tensorflow as tf
import tensorflow_datasets as tfds
Explanation: Tutorial: Using an input pipeline with datasets
In this example we will build training pipeline that uses tf.data.datasets for training the model.
End of explanation
(ds_train, ds_test), ds_info = tfds.load(
    'mnist',
    split=['train', 'test'],
    shuffle_files=True,
    as_supervised=True,
    with_info=True,
)
# tfds.load introduces a new logger which results in duplicate log messages.
# To mitigate this issue following removes Jupyter notebook root logger handler. More details @
# https://stackoverflow.com/questions/6729268/log-messages-appearing-twice-with-python-logging
import logging
logger = logging.getLogger()
logger.handlers = []
Explanation: Load MNIST Data
End of explanation
def normalize_img(image, label):
  Normalizes images: `uint8` -> `float32`.
  return tf.cast(image, tf.float32) / 255., label
ds_train = ds_train.map(
    normalize_img, num_parallel_calls=tf.data.experimental.AUTOTUNE)
ds_train = ds_train.cache()
ds_train = ds_train.shuffle(ds_info.splits['train'].num_examples)
ds_train = ds_train.batch(128)
ds_train = ds_train.prefetch(tf.data.experimental.AUTOTUNE)
ds_test = ds_test.map(
    normalize_img, num_parallel_calls=tf.data.experimental.AUTOTUNE)
ds_test = ds_test.batch(128)
ds_test = ds_test.cache()
ds_test = ds_test.prefetch(tf.data.experimental.AUTOTUNE)
Explanation: Build training pipeline
Build a training and evaluation pipeline using ds.map, ds.cache, ds.shuffle, ds.batch, and ds.prefetch. For more details on building high performance pipelines refer to data performance
End of explanation
def build_pipeline_model(hp):
    model = Sequential()
    model.add(Flatten(input_shape=(28, 28, 1)))
    # the number of layers is tunable
    for _ in range(hp.get('num_layers')):
        model.add(Dense(units=64, activation='relu'))
    model.add(Dense(10, activation='softmax'))
    # the learning rate is tunable
    model.compile(
        optimizer=Adam(lr=hp.get('learning_rate')),
        loss='sparse_categorical_crossentropy',
        metrics=['accuracy'])
    return model
# Configure the search space
pipeline_HPS = kerastuner.engine.hyperparameters.HyperParameters()
pipeline_HPS.Float('learning_rate', min_value=1e-4, max_value=1e-2, sampling='log')
pipeline_HPS.Int('num_layers', 2, 10)
pipeline_tuner = cloudtuner.CloudTuner(
    build_pipeline_model,
    project_id=PROJECT_ID,
    region=REGION,
    objective='accuracy',
    hyperparameters=pipeline_HPS,
    max_trials=5,
    directory='tmp_dir/2')
pipeline_tuner.search(x=ds_train, epochs=10, validation_data=ds_test)
pipeline_tuner.results_summary()
pipeline_model = pipeline_tuner.get_best_models(num_models=1)[0]
print(pipeline_model)
print(pipeline_model.weights)
Explanation: Create and train the model
End of explanation
# Configure the search space
STUDY_CONFIG = {
    'algorithm': 'ALGORITHM_UNSPECIFIED',
    'metrics': [{
        'goal': 'MAXIMIZE',
        'metric': 'accuracy'
    }],
    'parameters': [{
        'discrete_value_spec': {
            'values': [0.0001, 0.001, 0.01]
        },
        'parameter': 'learning_rate',
        'type': 'DISCRETE'
    }, {
        'integer_value_spec': {
            'max_value': 10,
            'min_value': 2
        },
        'parameter': 'num_layers',
        'type': 'INTEGER'
    }, {
        'discrete_value_spec': {
            'values': [32, 64, 96, 128]
        },
        'parameter': 'units',
        'type': 'DISCRETE'
    }],
    'automatedStoppingConfig': {
        'decayCurveStoppingConfig': {
            'useElapsedTime': True
        }
    }
}
Explanation: Tutorial: Using a Study Configuration
Now, let's repeat this study but this time the search space is passed in as a Optimizer study_config.
Create the Study Configuration
Let's start by constructing the study config for optimizing the accuracy of the model with the hyperparameters number of layers and learning rate, just as we did before.
End of explanation
tuner = cloudtuner.CloudTuner(
    build_model,
    project_id=PROJECT_ID,
    region=REGION,
    study_config=STUDY_CONFIG,
    max_trials=10,
    directory='tmp_dir/3')
Explanation: Instantiate CloudTuner
Next, we instantiate an instance of the CloudTuner. In this instantiation, we replace the hyperparameters and objective parameters with the study_config parameter.
End of explanation
tuner.search_space_summary()
Explanation: Let's use the search_space_summary() method to display what the search space for this optimization study looks like.
End of explanation
tuner.search(x=x, y=y, epochs=5, steps_per_epoch=2000, validation_steps=1000, validation_data=(val_x, val_y))
Explanation: Search
Let's now execute our search for this optimization study with the search() method.
End of explanation
tuner.results_summary()
Explanation: Results
Now let's use the results_summary() method to get a summary of the trials that were tried in this optimization study.
End of explanation
from multiprocessing.dummy import Pool
# If you are running this tutorial in a notebook locally, you may run multiple
# tuning loops concurrently using multi-processes instead of multi-threads.
# from multiprocessing import Pool
import time
import datetime
STUDY_ID = 'CloudTuner_study_{}'.format(
          datetime.datetime.now().strftime('%Y%m%d_%H%M%S'))
def single_tuner(tuner_id):
  Instantiate a `CloudTuner` and set up its `tuner_id`.
  Args:
    tuner_id: Integer.
  Returns:
    A CloudTuner.
  
  tuner = cloudtuner.CloudTuner(
        build_model,
        project_id=PROJECT_ID,
        region=REGION,
        objective='accuracy',
        hyperparameters=HPS,
        max_trials=18,
        study_id=STUDY_ID,
        directory=('tmp_dir/cloud/%s' % (STUDY_ID)))
  tuner.tuner_id = str(tuner_id)
  return tuner
def search_fn(tuner):
  # Start searching from different time points for each worker to avoid `model.build` collision.
  time.sleep(int(tuner.tuner_id)*2)
  tuner.search(x=x, y=y, epochs=5, validation_data=(val_x, val_y), verbose=0)
  return tuner
Explanation: Tutorial: Distributed Tuning
Let's run multiple tuning loops concurrently using multiple threads. To run distributed tuning, multiple tuners should share the same study_id, but different tuner_ids.
End of explanation
# Number of search loops we would like to run in parallel
num_parallel_trials = 4
tuners = [single_tuner(i) for i in range(num_parallel_trials)]
p = Pool(processes=num_parallel_trials)
result = p.map(search_fn, tuners)
p.close()
p.join()
Explanation: Search
Let's now execute multiple search loops in parallel for this study with the search() method.
End of explanation
result[0].results_summary()
Explanation: Results
Now let's use the results_summary() method to get a summary of the trials (from all the search loops) that were tried in this optimization study.
End of explanation
<END_TASK>