<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 E2E ML on GCP
Step1:  Restart the kernel
Once you've installed the additional packages, you need to restart the notebook kernel so it can find the packages.
Step2:  Before you begin
Set up your Google Cloud project
The following steps are required, regardless of your notebook environment.
Select or create a Google Cloud project. When you first create an account, you get a $300 free credit towards your compute/storage costs.
Make sure that billing is enabled for your project.
Enable the Vertex AI API. 
If you are running this notebook locally, you will need to install the Cloud SDK.
Enter your project ID in the cell below. Then run the cell to make sure the
Cloud SDK uses the right project for all the commands in this notebook.
Note
Step3:  Region
You can also change the REGION variable, which is used for operations
throughout the rest of this notebook.  Below are regions supported for Vertex AI. We recommend that you choose the region closest to you.
Americas
Step4:  Timestamp
If you are in a live tutorial session, you might be using a shared test account or project. To avoid name collisions between users on resources created, you create a timestamp for each instance session, and append the timestamp onto the name of resources you create in this tutorial.
Step5:  Authenticate your Google Cloud account
If you are using Vertex AI Workbench Notebooks, your environment is already authenticated. Skip this step.
If you are using Colab, run the cell below and follow the instructions when prompted to authenticate your account via oAuth.
Otherwise, follow these steps
Step6:  Create a Cloud Storage bucket
The following steps are required, regardless of your notebook environment.
When you initialize the Vertex SDK for Python, you specify a Cloud Storage staging bucket. The staging bucket is where all the data associated with your dataset and model resources are retained across sessions.
Set the name of your Cloud Storage bucket below. Bucket names must be globally unique across all Google Cloud projects, including those outside of your organization.
Step7:  Only if your bucket doesn't already exist
Step8:  Finally, validate access to your Cloud Storage bucket by examining its contents
Step9:  Set up variables
Next, set up some variables used throughout the tutorial.
Import libraries and define constants
Step10:  Initialize Vertex AI SDK for Python
Initialize the Vertex AI SDK for Python for your project and corresponding bucket.
Step11:  Set hardware accelerators
You can set hardware accelerators for training and prediction.
Set the variables TRAIN_GPU/TRAIN_NGPU and DEPLOY_GPU/DEPLOY_NGPU to use a container image supporting a GPU and the number of GPUs allocated to the virtual machine (VM) instance. For example, to use a GPU container image with 4 Nvidia Telsa K80 GPUs allocated to each VM, you would specify
Step12:  Set pre-built containers
Set the pre-built Docker container image for training and prediction.
For the latest list, see Pre-built containers for training.
For the latest list, see Pre-built containers for prediction.
Step13:  Set machine type
Next, set the machine type to use for training.
Set the variable TRAIN_COMPUTE to configure  the compute resources for the VMs you will use for for training.
machine type
n1-standard
Step14:  Introduction to scikit-learn training
Once you have trained a scikit-learn model, you will want to save it at a Cloud Storage location, so it can subsequently be uploaded to a Vertex AI Model resource. The Scikit-learn package does not have support to save the model to a Cloud Storage location. Instead, you will do the following steps to save to a Cloud Storage location.
Save the in-memory model to the local filesystem in pickle format (e.g., model.pkl).
Create a Cloud Storage storage client.
Upload the pickle file as a blob to the specified Cloud Storage location using the Cloud Storage storage client.
Note
Step17:  Create the task script for the Python training package
Next, you create the task.py script for driving the training package. Some noteable steps include
Step18:  Store training script on your Cloud Storage bucket
Next, you package the training folder into a compressed tar ball, and then store it in your Cloud Storage bucket.
Step19:  Create and run custom training job
To train a custom model, you perform two steps
Step20:  Prepare your command-line arguments
Now define the command-line arguments for your custom training container
Step21:  Run the custom training job
Next, you run the custom job to start the training job by invoking the method run, with the following parameters
Step22:  List a custom training job
Step23:  Wait for completion of custom training job
Next, wait for the custom training job to complete. Alternatively, one can set the parameter sync to True in the run() method to block until the custom training job is completed.
Step24:  Delete a custom training job
After a training job is completed, you can delete the training job with the method delete().  Prior to completion, a training job can be canceled with the method cancel().
Step25:  Cleaning up
To clean up all Google Cloud resources used in this project, you can delete the Google Cloud
project you used for the tutorial.
Otherwise, you can delete the individual resources you created in this tutorial
<ASSISTANT_TASK:>
Python Code:
import os
# The Vertex AI Workbench Notebook product has specific requirements
IS_WORKBENCH_NOTEBOOK = os.getenv("DL_ANACONDA_HOME")
IS_USER_MANAGED_WORKBENCH_NOTEBOOK = os.path.exists(
    "/opt/deeplearning/metadata/env_version"
)
# Vertex AI Notebook requires dependencies to be installed with '--user'
USER_FLAG = ""
if IS_WORKBENCH_NOTEBOOK:
    USER_FLAG = "--user"
! pip3 install {USER_FLAG} --upgrade google-cloud-aiplatform -q
Explanation: E2E ML on GCP: MLOps stage 2 : experimentation: get started with Vertex Training for Scikit-Learn
<table align="left">
  <td>
    <a href="https://colab.research.google.com/github/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/community/ml_ops/stage2/get_started_vertex_training_sklearn.ipynb">
      <img src="https://cloud.google.com/ml-engine/images/colab-logo-32px.png" alt="Colab logo"> Run in Colab
    </a>
  </td>
  <td>
    <a href="https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/community/ml_ops/stage2/get_started_vertex_training_sklearn.ipynb">
      <img src="https://cloud.google.com/ml-engine/images/github-logo-32px.png" alt="GitHub logo">
      View on GitHub
    </a>
  </td>
  <td>
    <a href="https://console.cloud.google.com/ai/platform/notebooks/deploy-notebook?download_url=https://raw.githubusercontent.com/GoogleCloudPlatform/vertex-ai-samples/main/notebooks/community/ml_ops/stage2/get_started_vertex_training_sklearn.ipynb">
     <img src="https://lh3.googleusercontent.com/UiNooY4LUgW_oTvpsNhPpQzsstV5W8F7rYgxgGBD85cWJoLmrOzhVs_ksK_vgx40SHs7jCqkTkCk=e14-rj-sc0xffffff-h130-w32" alt="Vertex AI logo">
Open in Vertex AI Workbench
    </a>
  </td>
</table>
&lt;/a&gt;
</td>
</table>
<br/><br/><br/>
Overview
This tutorial demonstrates how to use Vertex AI for E2E MLOps on Google Cloud in production. This tutorial covers stage 2 : experimentation: get started with Vertex AI Training for scikit-Learn.
Dataset
The dataset used for this tutorial is the News Aggregation from ICS Machine Learning Datasets. The trained model predicts the news category of the news article.
Objective
In this tutorial, you learn how to use Vertex AI Training for training a Scikit-Learn custom model.
This tutorial uses the following Google Cloud ML services:
Vertex AI Training
Vertex AI Model resource
The steps performed include:
Training using a Python package.
Report accuracy when hyperparameter tuning.
Save the model artifacts to Cloud Storage using GCSFuse.
Create a Vertex AI Model resource.
Costs
This tutorial uses billable components of Google Cloud:
Vertex AI
Cloud Storage
Learn about Vertex AI
pricing and Cloud Storage
pricing, and use the Pricing
Calculator
to generate a cost estimate based on your projected usage.
Set up your local development environment
If you are using Colab or Google Cloud Notebooks, your environment already meets
all the requirements to run this notebook. You can skip this step.
Otherwise, make sure your environment meets this notebook's requirements.
You need the following:
The Google Cloud SDK
Git
Python 3
virtualenv
Jupyter notebook running in a virtual environment with Python 3
The Google Cloud guide to Setting up a Python development
environment and the Jupyter
installation guide provide detailed instructions
for meeting these requirements. The following steps provide a condensed set of
instructions:
Install and initialize the Cloud SDK.
Install Python 3.
Install
   virtualenv
   and create a virtual environment that uses Python 3. Activate the virtual environment.
To install Jupyter, run pip3 install jupyter on the
command-line in a terminal shell.
To launch Jupyter, run jupyter notebook on the command-line in a terminal shell.
Open this notebook in the Jupyter Notebook Dashboard.
Install additional packages
Install the following packages for executing this notebook.
End of explanation
import os
if not os.getenv("IS_TESTING"):
    # Automatically restart kernel after installs
    import IPython
    app = IPython.Application.instance()
    app.kernel.do_shutdown(True)
Explanation: Restart the kernel
Once you've installed the additional packages, you need to restart the notebook kernel so it can find the packages.
End of explanation
PROJECT_ID = "[your-project-id]"  # @param {type:"string"}
if PROJECT_ID == "" or PROJECT_ID is None or PROJECT_ID == "[your-project-id]":
    # Get your GCP project id from gcloud
    shell_output = ! gcloud config list --format 'value(core.project)' 2>/dev/null
    PROJECT_ID = shell_output[0]
    print("Project ID:", PROJECT_ID)
! gcloud config set project $PROJECT_ID
Explanation: Before you begin
Set up your Google Cloud project
The following steps are required, regardless of your notebook environment.
Select or create a Google Cloud project. When you first create an account, you get a $300 free credit towards your compute/storage costs.
Make sure that billing is enabled for your project.
Enable the Vertex AI API. 
If you are running this notebook locally, you will need to install the Cloud SDK.
Enter your project ID in the cell below. Then run the cell to make sure the
Cloud SDK uses the right project for all the commands in this notebook.
Note: Jupyter runs lines prefixed with ! as shell commands, and it interpolates Python variables prefixed with $ into these commands.
Set your project ID
If you don't know your project ID, you may be able to get your project ID using gcloud.
End of explanation
REGION = "[your-region]"  # @param {type: "string"}
if REGION == "[your-region]":
    REGION = "us-central1"
Explanation: Region
You can also change the REGION variable, which is used for operations
throughout the rest of this notebook.  Below are regions supported for Vertex AI. We recommend that you choose the region closest to you.
Americas: us-central1
Europe: europe-west4
Asia Pacific: asia-east1
You may not use a multi-regional bucket for training with Vertex AI. Not all regions provide support for all Vertex AI services.
Learn more about Vertex AI regions.
End of explanation
from datetime import datetime
TIMESTAMP = datetime.now().strftime("%Y%m%d%H%M%S")
Explanation: Timestamp
If you are in a live tutorial session, you might be using a shared test account or project. To avoid name collisions between users on resources created, you create a timestamp for each instance session, and append the timestamp onto the name of resources you create in this tutorial.
End of explanation
# If you are running this notebook in Colab, run this cell and follow the
# instructions to authenticate your GCP account. This provides access to your
# Cloud Storage bucket and lets you submit training jobs and prediction
# requests.
import os
import sys
# If on Vertex AI Workbench, then don't execute this code
IS_COLAB = False
if not os.path.exists("/opt/deeplearning/metadata/env_version") and not os.getenv(
    "DL_ANACONDA_HOME"
):
    if "google.colab" in sys.modules:
        IS_COLAB = True
        from google.colab import auth as google_auth
        google_auth.authenticate_user()
    # If you are running this notebook locally, replace the string below with the
    # path to your service account key and run this cell to authenticate your GCP
    # account.
    elif not os.getenv("IS_TESTING"):
        %env GOOGLE_APPLICATION_CREDENTIALS ''
Explanation: Authenticate your Google Cloud account
If you are using Vertex AI Workbench Notebooks, your environment is already authenticated. Skip this step.
If you are using Colab, run the cell below and follow the instructions when prompted to authenticate your account via oAuth.
Otherwise, follow these steps:
In the Cloud Console, go to the Create service account key page.
Click Create service account.
In the Service account name field, enter a name, and click Create.
In the Grant this service account access to project section, click the Role drop-down list. Type "Vertex AI" into the filter box, and select Vertex AI Administrator. Type "Storage Object Admin" into the filter box, and select Storage Object Admin.
Click Create. A JSON file that contains your key downloads to your local environment.
Enter the path to your service account key as the GOOGLE_APPLICATION_CREDENTIALS variable in the cell below and run the cell.
End of explanation
BUCKET_URI = "gs://[your-bucket-name]"  # @param {type:"string"}
if BUCKET_URI == "" or BUCKET_URI is None or BUCKET_URI == "gs://[your-bucket-name]":
    BUCKET_URI = "gs://" + PROJECT_ID + "aip-" + TIMESTAMP
Explanation: Create a Cloud Storage bucket
The following steps are required, regardless of your notebook environment.
When you initialize the Vertex SDK for Python, you specify a Cloud Storage staging bucket. The staging bucket is where all the data associated with your dataset and model resources are retained across sessions.
Set the name of your Cloud Storage bucket below. Bucket names must be globally unique across all Google Cloud projects, including those outside of your organization.
End of explanation
! gsutil mb -l $REGION $BUCKET_URI
Explanation: Only if your bucket doesn't already exist: Run the following cell to create your Cloud Storage bucket.
End of explanation
! gsutil ls -al $BUCKET_URI
Explanation: Finally, validate access to your Cloud Storage bucket by examining its contents:
End of explanation
import google.cloud.aiplatform as aip
Explanation: Set up variables
Next, set up some variables used throughout the tutorial.
Import libraries and define constants
End of explanation
aip.init(project=PROJECT_ID, staging_bucket=BUCKET_URI)
Explanation: Initialize Vertex AI SDK for Python
Initialize the Vertex AI SDK for Python for your project and corresponding bucket.
End of explanation
import os
if os.getenv("IS_TESTING_TRAIN_GPU"):
    TRAIN_GPU, TRAIN_NGPU = (
        aip.gapic.AcceleratorType.NVIDIA_TESLA_K80,
        int(os.getenv("IS_TESTING_TRAIN_GPU")),
    )
else:
    TRAIN_GPU, TRAIN_NGPU = (None, None)
if os.getenv("IS_TESTING_DEPLOY_GPU"):
    DEPLOY_GPU, DEPLOY_NGPU = (
        aip.gapic.AcceleratorType.NVIDIA_TESLA_K80,
        int(os.getenv("IS_TESTING_DEPLOY_GPU")),
    )
else:
    DEPLOY_GPU, DEPLOY_NGPU = (None, None)
Explanation: Set hardware accelerators
You can set hardware accelerators for training and prediction.
Set the variables TRAIN_GPU/TRAIN_NGPU and DEPLOY_GPU/DEPLOY_NGPU to use a container image supporting a GPU and the number of GPUs allocated to the virtual machine (VM) instance. For example, to use a GPU container image with 4 Nvidia Telsa K80 GPUs allocated to each VM, you would specify:
(aip.AcceleratorType.NVIDIA_TESLA_K80, 4)
Otherwise specify (None, None) to use a container image to run on a CPU.
Learn more about hardware accelerator support for your region.
Note: TF releases before 2.3 for GPU support will fail to load the custom model in this tutorial. It is a known issue and fixed in TF 2.3. This is caused by static graph ops that are generated in the serving function. If you encounter this issue on your own custom models, use a container image for TF 2.3 with GPU support.
End of explanation
TRAIN_VERSION = "scikit-learn-cpu.0-23"
DEPLOY_VERSION = "sklearn-cpu.0-23"
TRAIN_IMAGE = "{}-docker.pkg.dev/vertex-ai/training/{}:latest".format(
    REGION.split("-")[0], TRAIN_VERSION
)
DEPLOY_IMAGE = "{}-docker.pkg.dev/vertex-ai/prediction/{}:latest".format(
    REGION.split("-")[0], DEPLOY_VERSION
)
Explanation: Set pre-built containers
Set the pre-built Docker container image for training and prediction.
For the latest list, see Pre-built containers for training.
For the latest list, see Pre-built containers for prediction.
End of explanation
if os.getenv("IS_TESTING_TRAIN_MACHINE"):
    MACHINE_TYPE = os.getenv("IS_TESTING_TRAIN_MACHINE")
else:
    MACHINE_TYPE = "n1-standard"
VCPU = "4"
TRAIN_COMPUTE = MACHINE_TYPE + "-" + VCPU
print("Train machine type", TRAIN_COMPUTE)
Explanation: Set machine type
Next, set the machine type to use for training.
Set the variable TRAIN_COMPUTE to configure  the compute resources for the VMs you will use for for training.
machine type
n1-standard: 3.75GB of memory per vCPU.
n1-highmem: 6.5GB of memory per vCPU
n1-highcpu: 0.9 GB of memory per vCPU
vCPUs: number of [2, 4, 8, 16, 32, 64, 96 ]
Note: The following is not supported for training:
standard: 2 vCPUs
highcpu: 2, 4 and 8 vCPUs
Note: You may also use n2 and e2 machine types for training and deployment, but they do not support GPUs.
End of explanation
# Make folder for Python training script
! rm -rf custom
! mkdir custom
# Add package information
! touch custom/README.md
setup_cfg = "[egg_info]\n\ntag_build =\n\ntag_date = 0"
! echo "$setup_cfg" > custom/setup.cfg
setup_py = "import setuptools\n\nsetuptools.setup(\n\n    install_requires=[\n\n        'wget',\n\n        'cloudml-hypertune',\n\n    ],\n\n    packages=setuptools.find_packages())"
! echo "$setup_py" > custom/setup.py
pkg_info = "Metadata-Version: 1.0\n\nName: News Aggregation text classification\n\nVersion: 0.0.0\n\nSummary: Demostration training script\n\nHome-page: www.google.com\n\nAuthor: Google\n\nAuthor-email: aferlitsch@google.com\n\nLicense: Public\n\nDescription: Demo\n\nPlatform: Vertex"
! echo "$pkg_info" > custom/PKG-INFO
# Make the training subfolder
! mkdir custom/trainer
! touch custom/trainer/__init__.py
Explanation: Introduction to scikit-learn training
Once you have trained a scikit-learn model, you will want to save it at a Cloud Storage location, so it can subsequently be uploaded to a Vertex AI Model resource. The Scikit-learn package does not have support to save the model to a Cloud Storage location. Instead, you will do the following steps to save to a Cloud Storage location.
Save the in-memory model to the local filesystem in pickle format (e.g., model.pkl).
Create a Cloud Storage storage client.
Upload the pickle file as a blob to the specified Cloud Storage location using the Cloud Storage storage client.
Note: You can do hyperparameter tuning with a Scikit-learn model.
Examine the training package
Package layout
Before you start the training, you will look at how a Python package is assembled for a custom training job. When unarchived, the package contains the following directory/file layout.
PKG-INFO
README.md
setup.cfg
setup.py
trainer
__init__.py
task.py
The files setup.cfg and setup.py are the instructions for installing the package into the operating environment of the Docker image.
The file trainer/task.py is the Python script for executing the custom training job. Note, when we referred to it in the worker pool specification, we replace the directory slash with a dot (trainer.task) and dropped the file suffix (.py).
Package Assembly
In the following cells, you will assemble the training package.
End of explanation
%%writefile custom/trainer/task.py
import argparse
import logging
import os
import pickle
import zipfile
from typing import List, Tuple
import pandas as pd
import wget
from sklearn.feature_extraction.text import CountVectorizer, TfidfTransformer
from sklearn.model_selection import train_test_split
from sklearn.naive_bayes import MultinomialNB
from sklearn.pipeline import Pipeline
import hypertune
parser = argparse.ArgumentParser()
parser.add_argument('--model-dir', dest='model_dir',
                    default=os.getenv('AIP_MODEL_DIR'), type=str, help='Model dir.')
parser.add_argument("--dataset-url", dest="dataset_url",
                    type=str, help="Download url for the training data.")
parser.add_argument('--alpha', dest='alpha',
                    default=1.0, type=float,
                    help='Alpha parameters for MultinomialNB')
args = parser.parse_args()
logging.getLogger().setLevel(logging.INFO)
def get_data(url: str, test_size: float = 0.2) -> Tuple[List, List, List, List]:
    logging.info("Downloading training data from: {}".format(args.dataset_url))
    zip_filepath = wget.download(url, out=".")
    with zipfile.ZipFile(zip_filepath, "r") as zf:
        zf.extract(path=".", member="newsCorpora.csv")
    COLUMN_NAMES = ["id", "title", "url", "publisher",
                    "category", "story", "hostname", "timestamp"]
    dataframe = pd.read_csv(
        "newsCorpora.csv", delimiter="	", names=COLUMN_NAMES, index_col=0
    )
    train, test = train_test_split(dataframe, test_size=test_size)
    x_train, y_train = train["title"].values, train["category"].values
    x_test, y_test = test["title"].values, test["category"].values
    return x_train, y_train, x_test, y_test
def get_model():
    logging.info("Build model ...")
    model = Pipeline([
                ("vectorizer", CountVectorizer()),
                ("tfidf", TfidfTransformer()),
                ("naivebayes", MultinomialNB(alpha=args.alpha)),
        ])
    return model
def train_model(model: Pipeline, X_train: List, y_train: List, X_test: List, y_test: List
) -> Pipeline:
    logging.info("Training started ...")
    model.fit(X_train, y_train)
    logging.info("Training completed")
    return model
def evaluate_model(model: Pipeline, X_train: List, y_train: List, X_test: List, y_test: List
) -> float:
    score = model.score(X_test, y_test)
    logging.info(f"Evaluation completed with model score: {score}")
    # report metric for hyperparameter tuning
    hpt = hypertune.HyperTune()
    hpt.report_hyperparameter_tuning_metric(
        hyperparameter_metric_tag='accuracy',
        metric_value=score
    )
    return score
def export_model_to_gcs(fitted_pipeline: Pipeline, gcs_uri: str) -> str:
    Exports trained pipeline to GCS
    Parameters:
            fitted_pipeline (sklearn.pipelines.Pipeline): the Pipeline object
                with data already fitted (trained pipeline object).
            gcs_uri (str): GCS path to store the trained pipeline
                i.e gs://example_bucket/training-job.
    Returns:
            export_path (str): Model GCS location
    
    # Upload model artifact to Cloud Storage
    artifact_filename = 'model.pkl'
    storage_path = os.path.join(gcs_uri, artifact_filename)
    # Save model artifact to local filesystem (doesn't persist)
    with open(storage_path, 'wb') as model_file:
        pickle.dump(fitted_pipeline, model_file)
def export_evaluation_report_to_gcs(report: str, gcs_uri: str) -> None:
    
    Exports training job report to GCS
    Parameters:
        report (str): Full report in text to sent to GCS
        gcs_uri (str): GCS path to store the report
            i.e gs://example_bucket/training-job
    
    # Upload model artifact to Cloud Storage
    artifact_filename = 'report.txt'
    storage_path = os.path.join(gcs_uri, artifact_filename)
    # Save model artifact to local filesystem (doesn't persist)
    with open(storage_path, 'w') as report_file:
        report_file.write(report)
logging.info("Starting custom training job.")
data = get_data(args.dataset_url)
model = get_model()
model = train_model(model, *data)
score = evaluate_model(model, *data)
# export model to gcs using GCSFuse
logging.info("Exporting model artifacts ...")
gs_prefix = 'gs://'
gcsfuse_prefix = '/gcs/'
if args.model_dir.startswith(gs_prefix):
    args.model_dir = args.model_dir.replace(gs_prefix, gcsfuse_prefix)
    dirpath = os.path.split(args.model_dir)[0]
    if not os.path.isdir(dirpath):
        os.makedirs(dirpath)
export_model_to_gcs(model, args.model_dir)
export_evaluation_report_to_gcs(str(score), args.model_dir)
logging.info(f"Exported model artifacts to GCS bucket: {args.model_dir}")
Explanation: Create the task script for the Python training package
Next, you create the task.py script for driving the training package. Some noteable steps include:
Command-line arguments:
model-dir: The location to save the trained model. When using Vertex AI custom training, the location will be specified in the environment variable: AIP_MODEL_DIR,
dataset_url: The location of the dataset to download.
alpha: Hyperparameter
Data preprocessing (get_data()):
Download the dataset and split into training and test.
Model architecture (get_model()):
Builds the corresponding model architecture.
Training (train_model()):
Trains the model
Evaluation (evaluate_model()):
Evaluates the model.
If hyperparameter tuning, reports the metric for accuracy.
Model artifact saving
Saves the model artifacts and evaluation metrics where the Cloud Storage location specified by model-dir.
Note: GCSFuse (/gcs) is used to do filesystem operations on Cloud Storage buckets.
End of explanation
! rm -f custom.tar custom.tar.gz
! tar cvf custom.tar custom
! gzip custom.tar
! gsutil cp custom.tar.gz $BUCKET_URI/trainer_newsaggr.tar.gz
Explanation: Store training script on your Cloud Storage bucket
Next, you package the training folder into a compressed tar ball, and then store it in your Cloud Storage bucket.
End of explanation
DISPLAY_NAME = "newsaggr_" + TIMESTAMP
job = aip.CustomPythonPackageTrainingJob(
    display_name=DISPLAY_NAME,
    python_package_gcs_uri=f"{BUCKET_URI}/trainer_newsaggr.tar.gz",
    python_module_name="trainer.task",
    container_uri=TRAIN_IMAGE,
    model_serving_container_image_uri=DEPLOY_IMAGE,
    project=PROJECT_ID,
)
Explanation: Create and run custom training job
To train a custom model, you perform two steps: 1) create a custom training job, and 2) run the job.
Create custom training job
A custom training job is created with the CustomTrainingJob class, with the following parameters:
display_name: The human readable name for the custom training job.
container_uri: The training container image.
python_package_gcs_uri: The location of the Python training package as a tarball.
python_module_name: The relative path to the training script in the Python package.
model_serving_container_uri: The container image for deploying the model.
Note: There is no requirements parameter. You specify any requirements in the setup.py script in your Python package.
End of explanation
MODEL_DIR = "{}/{}".format(BUCKET_URI, TIMESTAMP)
DATASET_URL = "https://archive.ics.uci.edu/ml/machine-learning-databases/00359/NewsAggregatorDataset.zip"
DIRECT = False
if DIRECT:
    CMDARGS = [
        "--alpha=" + str(0.9),
        "--dataset-url=" + DATASET_URL,
        "--model_dir=" + MODEL_DIR,
    ]
else:
    CMDARGS = ["--alpha=" + str(0.9), "--dataset-url=" + DATASET_URL]
Explanation: Prepare your command-line arguments
Now define the command-line arguments for your custom training container:
args: The command-line arguments to pass to the executable that is set as the entry point into the container.
--model-dir : For our demonstrations, we use this command-line argument to specify where to store the model artifacts.
direct: You pass the Cloud Storage location as a command line argument to your training script (set variable DIRECT = True), or
indirect: The service passes the Cloud Storage location as the environment variable AIP_MODEL_DIR to your training script (set variable DIRECT = False). In this case, you tell the service the model artifact location in the job specification.
--dataset-url: The location of the dataset to download.
--alpha: Tunable hyperparameter
End of explanation
if TRAIN_GPU:
    model = job.run(
        model_display_name="newsaggr_" + TIMESTAMP,
        args=CMDARGS,
        replica_count=1,
        machine_type=TRAIN_COMPUTE,
        accelerator_type=TRAIN_GPU.name,
        accelerator_count=TRAIN_NGPU,
        base_output_dir=MODEL_DIR,
        sync=False,
    )
else:
    model = job.run(
        model_display_name="newsaggr_" + TIMESTAMP,
        args=CMDARGS,
        replica_count=1,
        machine_type=TRAIN_COMPUTE,
        base_output_dir=MODEL_DIR,
        sync=False,
    )
model_path_to_deploy = MODEL_DIR
Explanation: Run the custom training job
Next, you run the custom job to start the training job by invoking the method run, with the following parameters:
model_display_name: The human readable name for the Model resource.
args: The command-line arguments to pass to the training script.
replica_count: The number of compute instances for training (replica_count = 1 is single node training).
machine_type: The machine type for the compute instances.
accelerator_type: The hardware accelerator type.
accelerator_count: The number of accelerators to attach to a worker replica.
base_output_dir: The Cloud Storage location to write the model artifacts to.
sync: Whether to block until completion of the job.
End of explanation
_job = job.list(filter=f"display_name={DISPLAY_NAME}")
print(_job)
Explanation: List a custom training job
End of explanation
model.wait()
Explanation: Wait for completion of custom training job
Next, wait for the custom training job to complete. Alternatively, one can set the parameter sync to True in the run() method to block until the custom training job is completed.
End of explanation
job.delete()
Explanation: Delete a custom training job
After a training job is completed, you can delete the training job with the method delete().  Prior to completion, a training job can be canceled with the method cancel().
End of explanation
# Delete the model using the Vertex model object
model.delete()
delete_bucket = False
if delete_bucket or os.getenv("IS_TESTING"):
    ! gsutil rm -r $BUCKET_URI
Explanation: Cleaning up
To clean up all Google Cloud resources used in this project, you can delete the Google Cloud
project you used for the tutorial.
Otherwise, you can delete the individual resources you created in this tutorial:
Model
Cloud Storage Bucket
End of explanation
<END_TASK>