<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 By deploying or using this software you agree to comply with the AI Hub Terms of Service and the Google APIs Terms of Service. To the extent of a direct conflict of terms, the AI Hub Terms of Service will control.
Overview
This notebook provides an example workflow of using the Tabular data inspection for a visual inspection of datasets.
Dataset
The notebook uses the Boston housing price regression dataset. It containers 506 observations with 13 features describing a house in Boston and a corresponding house price, stored in a 506x14 table.
Objective
The goal of this notebook is to go through a common training workflow
Step1:  Authenticate your GCP account
If you are using AI Platform Notebooks, your environment is already
authenticated. Skip this step.
If you are using Colab, run the cell below and follow the instructions
when prompted to authenticate your account via oAuth.
Otherwise, follow these steps
Step2:  Create a Cloud Storage bucket
The following steps are required, regardless of your notebook environment.
You need to have a "workspace" bucket that will hold the dataset and the output from the ML Container. Set the name of your Cloud Storage bucket below. It must be unique across all Cloud Storage buckets. 
You may also change the REGION variable, which is used for operations
throughout the rest of this notebook. Make sure to choose a region where Cloud AI Platform services are available. You may not use a Multi-Regional Storage bucket for training with AI Platform.
Step3:  Only if your bucket doesn't already exist
Step4:  Finally, validate access to your Cloud Storage bucket by examining its contents
Step5:  Import libraries and define constants
Step6:  Create a dataset
Step7:  Cloud Run
Accelerator and distribution support
| GPU | Multi-GPU Node | TPU | Workers | Parameter Server |
|---|---|---|---|---|
| No | No | No | No | No |
AI Platform training
Tabular data inspection.
AI Platform training documentation.
Local Run
Step8:  Local training snippet
Note that the training can also be done locally with Docker
bash
docker run \
    -v /tmp
Step9:  Cleaning up
To clean up all GCP resources used in this project, you can delete the GCP
project you used for the tutorial.
<ASSISTANT_TASK:>
Python Code:
PROJECT_ID = "[your-project-id]" #@param {type:"string"}
! gcloud config set project $PROJECT_ID
Explanation: By deploying or using this software you agree to comply with the AI Hub Terms of Service and the Google APIs Terms of Service. To the extent of a direct conflict of terms, the AI Hub Terms of Service will control.
Overview
This notebook provides an example workflow of using the Tabular data inspection for a visual inspection of datasets.
Dataset
The notebook uses the Boston housing price regression dataset. It containers 506 observations with 13 features describing a house in Boston and a corresponding house price, stored in a 506x14 table.
Objective
The goal of this notebook is to go through a common training workflow:
- Create a dataset
- Use AI Platform Training service to create a visual "Run Report" from the dataset
- Inspect the dataset by looking at the generated "Run Report"
Costs
This tutorial uses billable components of Google Cloud Platform (GCP):
Cloud AI Platform
Cloud Storage
Learn about Cloud AI Platform
pricing and Cloud Storage
pricing, and use the Pricing
Calculator
to generate a cost estimate based on your projected usage.
Set up your local development environment
If you are using Colab or AI Platform Notebooks, your environment already meets
all the requirements to run this notebook. You can skip this step.
Otherwise, make sure your environment meets this notebook's requirements.
You need the following:
The Google Cloud SDK
Git
Python 3
virtualenv
Jupyter notebook running in a virtual environment with Python 3
The Google Cloud guide to Setting up a Python development
environment and the Jupyter
installation guide provide detailed instructions
for meeting these requirements. The following steps provide a condensed set of
instructions:
Install and initialize the Cloud SDK.
Install Python 3.
Install
   virtualenv
   and create a virtual environment that uses Python 3.
Activate that environment and run pip install jupyter in a shell to install
   Jupyter.
Run jupyter notebook in a shell to launch Jupyter.
Open this notebook in the Jupyter Notebook Dashboard.
Set up your GCP project
The following steps are required, regardless of your notebook environment.
Select or create a GCP project.. When you first create an account, you get a $300 free credit towards your compute/storage costs.
Make sure that billing is enabled for your project.
Enable the AI Platform APIs and Compute Engine APIs.
Enter your project ID in the cell below. Then run the  cell to make sure the
Cloud SDK uses the right project for all the commands in this notebook.
Note: Jupyter runs lines prefixed with ! as shell commands, and it interpolates Python variables prefixed with $ into these commands.
End of explanation
import sys
# If you are running this notebook in Colab, run this cell and follow the
# instructions to authenticate your GCP account. This provides access to your
# Cloud Storage bucket and lets you submit training jobs and prediction
# requests.
if 'google.colab' in sys.modules:
  from google.colab import auth as google_auth
  google_auth.authenticate_user()
# If you are running this notebook locally, replace the string below with the
# path to your service account key and run this cell to authenticate your GCP
# account.
else:
  %env GOOGLE_APPLICATION_CREDENTIALS ''
Explanation: Authenticate your GCP account
If you are using AI Platform Notebooks, your environment is already
authenticated. Skip this step.
If you are using Colab, run the cell below and follow the instructions
when prompted to authenticate your account via oAuth.
Otherwise, follow these steps:
In the GCP Console, go to the Create service account key
   page.
From the Service account drop-down list, select New service account.
In the Service account name field, enter a name.
From the Role drop-down list, select
   Machine Learning Engine > AI Platform Admin and
   Storage > Storage Object Admin.
Click Create. A JSON file that contains your key downloads to your
local environment.
Enter the path to your service account key as the
GOOGLE_APPLICATION_CREDENTIALS variable in the cell below and run the cell.
End of explanation
BUCKET_NAME = "[your-bucket-name]" #@param {type:"string"}
REGION = 'us-central1' #@param {type:"string"}
Explanation: Create a Cloud Storage bucket
The following steps are required, regardless of your notebook environment.
You need to have a "workspace" bucket that will hold the dataset and the output from the ML Container. Set the name of your Cloud Storage bucket below. It must be unique across all Cloud Storage buckets. 
You may also change the REGION variable, which is used for operations
throughout the rest of this notebook. Make sure to choose a region where Cloud AI Platform services are available. You may not use a Multi-Regional Storage bucket for training with AI Platform.
End of explanation
! gsutil mb -l $REGION gs://$BUCKET_NAME
Explanation: Only if your bucket doesn't already exist: Run the following cell to create your Cloud Storage bucket.
End of explanation
! gsutil ls -al gs://$BUCKET_NAME
Explanation: Finally, validate access to your Cloud Storage bucket by examining its contents:
End of explanation
from __future__ import absolute_import
from __future__ import division
from __future__ import print_function
import os
import time
import pandas as pd
import tensorflow as tf
from IPython.core.display import HTML
Explanation: Import libraries and define constants
End of explanation
bh = tf.keras.datasets.boston_housing
(X_train, y_train), (X_eval, y_eval) = bh.load_data()
training = pd.DataFrame(X_train)
training['target'] = y_train
validation = pd.DataFrame(X_eval)
validation['target'] = y_eval
print('Data head:')
display(training.head(2))
data = os.path.join('gs://', BUCKET_NAME, 'data.csv')
print('Copy the data in bucket ...')
with tf.io.gfile.GFile(data, 'w') as f:
  training.append(validation).to_csv(f, index=False)
Explanation: Create a dataset
End of explanation
output_location = os.path.join('gs://', BUCKET_NAME, 'output')
job_name = "data_inspection_{}".format(time.strftime("%Y%m%d%H%M%S"))
!gcloud ai-platform jobs submit training $job_name \
    --master-image-uri gcr.io/aihub-c2t-containers/kfp-components/oob_algorithm/tabular_data_inspection:latest \
    --region $REGION \
    --scale-tier CUSTOM \
    --master-machine-type standard \
    -- \
    --output-location {output_location} \
    --data {data} \
    --data-type csv
Explanation: Cloud Run
Accelerator and distribution support
| GPU | Multi-GPU Node | TPU | Workers | Parameter Server |
|---|---|---|---|---|
| No | No | No | No | No |
AI Platform training
Tabular data inspection.
AI Platform training documentation.
Local Run
End of explanation
if not tf.io.gfile.exists(os.path.join(output_location, 'report.html')):
  raise RuntimeError('The file report.html was not found. Did the training job finish?')
with tf.io.gfile.GFile(os.path.join(output_location, 'report.html')) as f:
  display(HTML(f.read()))
Explanation: Local training snippet
Note that the training can also be done locally with Docker
bash
docker run \
    -v /tmp:/tmp \
    -it gcr.io/aihub-c2t-containers/kfp-components/oob_algorithm/tabular_data_inspection:latest \
    --output-location /tmp/tabular_data_inspection \
    --data /tmp/data.csv \
    --data-type csv
Inspect the Run Report
The "Run Report" will help you identify if the model was successfully trained.
End of explanation
# If training job is still running, cancel it
! gcloud ai-platform jobs cancel $job_name --quiet
# Delete Cloud Storage objects that were created
! gsutil -m rm -r $BUCKET_NAME
Explanation: Cleaning up
To clean up all GCP resources used in this project, you can delete the GCP
project you used for the tutorial.
End of explanation
<END_TASK>