<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 Continuous training with TFX and Google Cloud AI Platform
Learning Objectives
Use the TFX CLI to build a TFX pipeline.
Deploy a TFX pipeline version with tuning enabled to a hosted AI Platform Pipelines instance.
Create and monitor a TFX pipeline run using the TFX CLI and KFP UI.
In this lab, you use utilize the following tools and services to deploy and run a TFX pipeline on Google Cloud that automates the development and deployment of a TensorFlow 2.3 WideDeep Classifer to predict forest cover from cartographic data
Step1:  Validate lab package version installation
Step2:  Note
Step3:  Note
Step4:  The config.py module configures the default values for the environment specific settings and the default values for the pipeline runtime parameters. 
The default values can be overwritten at compile time by providing the updated values in a set of environment variables updated in this lab notebook below.
The pipeline.py module contains the TFX DSL defining the workflow implemented by the pipeline.
The preprocessing.py module implements the data preprocessing logic  the Transform component.
The model.py module implements the training, tuning, and model building logic for the Trainer and Tuner components.
The runner.py module configures and executes KubeflowDagRunner. At compile time, the KubeflowDagRunner.run() method converts the TFX DSL into the pipeline package in the argo format for execution on your AI Platform Pipelines instance.
The features.py module contains feature definitions common across preprocessing.py and model.py.
Exercise
Step5:  Set the pipeline compile time settings
Default pipeline runtime environment values are configured in the pipeline folder config.py. You will set their values directly below
Step6:  Compile your pipeline code
You can build and upload the pipeline to the AI Platform Pipelines instance in one step, using the tfx pipeline create command. The tfx pipeline create goes through the following steps
Step7:  Note
Step8:  If you make a mistake above and need to redeploy the pipeline you can first delete the previous version using tfx pipeline delete or you can update the pipeline in-place using tfx pipeline update.
To delete the pipeline
Step9:  Exercise
Step10:  To retrieve the status of a given run retrieved from the command above
<ASSISTANT_TASK:>
Python Code:
import yaml
# Set `PATH` to include the directory containing TFX CLI and skaffold.
PATH=%env PATH
%env PATH=/home/jupyter/.local/bin:{PATH}
Explanation: Continuous training with TFX and Google Cloud AI Platform
Learning Objectives
Use the TFX CLI to build a TFX pipeline.
Deploy a TFX pipeline version with tuning enabled to a hosted AI Platform Pipelines instance.
Create and monitor a TFX pipeline run using the TFX CLI and KFP UI.
In this lab, you use utilize the following tools and services to deploy and run a TFX pipeline on Google Cloud that automates the development and deployment of a TensorFlow 2.3 WideDeep Classifer to predict forest cover from cartographic data:
The TFX CLI utility to build and deploy a TFX pipeline.
A hosted AI Platform Pipeline instance (Kubeflow Pipelines) for TFX pipeline orchestration.
Dataflow jobs for scalable, distributed data processing for TFX components.
A AI Platform Training job for model training and flock management of tuning trials. 
AI Platform Prediction, a model server destination for blessed pipeline model versions.
CloudTuner (KerasTuner implementation) and AI Platform Vizier for advanced model hyperparameter tuning.
You will then create and monitor pipeline runs using the TFX CLI as well as the KFP UI.
Setup
Update lab environment PATH to include TFX CLI and skaffold
End of explanation
!python -c "import tensorflow; print('TF version: {}'.format(tensorflow.__version__))"
!python -c "import tfx; print('TFX version: {}'.format(tfx.__version__))"
!python -c "import kfp; print('KFP version: {}'.format(kfp.__version__))"
Explanation: Validate lab package version installation
End of explanation
%pip install --upgrade --user tensorflow==2.3.2
%pip install --upgrade --user tfx==0.25.0
%pip install --upgrade --user kfp==1.4.0
Explanation: Note: this lab was built and tested with the following package versions:
TF version: 2.3.2
TFX version: 0.25.0
KFP version: 1.4.0
(Optional) If running the above command results in different package versions or you receive an import error, upgrade to the correct versions by running the cell below:
End of explanation
%cd pipeline
!ls -la
Explanation: Note: you may need to restart the kernel to pick up the correct package versions.
Validate creation of AI Platform Pipelines cluster
Navigate to AI Platform Pipelines page in the Google Cloud Console.
Note you may have already deployed an AI Pipelines instance during the Setup for the lab series. If so, you can proceed using that instance. If not:
Create or select an existing Kubernetes cluster (GKE) and deploy AI Platform. Make sure to select "Allow access to the following Cloud APIs https://www.googleapis.com/auth/cloud-platform" to allow for programmatic access to your pipeline by the Kubeflow SDK for the rest of the lab. Also, provide an App instance name such as "tfx" or "mlops". 
Validate the deployment of your AI Platform Pipelines instance in the console before proceeding.
Exercise: review the example TFX pipeline design pattern for Google Cloud
The pipeline source code can be found in the pipeline folder.
End of explanation
PROJECT_ID = !(gcloud config get-value core/project)
PROJECT_ID = PROJECT_ID[0]
GCP_REGION = 'us-central1'
ARTIFACT_STORE_URI = f'gs://{PROJECT_ID}-kubeflowpipelines-default'
CUSTOM_SERVICE_ACCOUNT = f'tfx-tuner-caip-service-account@{PROJECT_ID}.iam.gserviceaccount.com'
#TODO: Set your environment resource settings here for ENDPOINT.
ENDPOINT = ''
# Set your resource settings as Python environment variables. These override the default values in pipeline/config.py.
%env GCP_REGION={GCP_REGION}
%env ARTIFACT_STORE_URI={ARTIFACT_STORE_URI}
%env CUSTOM_SERVICE_ACCOUNT={CUSTOM_SERVICE_ACCOUNT}
%env PROJECT_ID={PROJECT_ID}
Explanation: The config.py module configures the default values for the environment specific settings and the default values for the pipeline runtime parameters. 
The default values can be overwritten at compile time by providing the updated values in a set of environment variables updated in this lab notebook below.
The pipeline.py module contains the TFX DSL defining the workflow implemented by the pipeline.
The preprocessing.py module implements the data preprocessing logic  the Transform component.
The model.py module implements the training, tuning, and model building logic for the Trainer and Tuner components.
The runner.py module configures and executes KubeflowDagRunner. At compile time, the KubeflowDagRunner.run() method converts the TFX DSL into the pipeline package in the argo format for execution on your AI Platform Pipelines instance.
The features.py module contains feature definitions common across preprocessing.py and model.py.
Exercise: build your pipeline package with the TFX CLI
You will use TFX CLI to compile and deploy the pipeline. As explained in the previous section, the environment specific settings can be provided through a set of environment variables and embedded into the pipeline package at compile time.
Configure your environment resource settings
Update  the below constants  with the settings reflecting your lab environment. 
GCP_REGION - the compute region for AI Platform Training, Vizier, and Prediction.
ARTIFACT_STORE - An existing GCS bucket. You can use any bucket or use the GCS bucket created during installation of AI Platform Pipelines. The default bucket name will contain the kubeflowpipelines-default prefix.
CUSTOM_SERVICE_ACCOUNT - In the gcp console Click on the Navigation Menu. Navigate to IAM &amp; Admin, then to Service Accounts and use the service account starting with prefix - 'tfx-tuner-caip-service-account'. This enables CloudTuner and the Google Cloud AI Platform extensions Tuner component to work together and allows for distributed and parallel tuning backed by AI Platform Vizier's hyperparameter search algorithm.
ENDPOINT - set the ENDPOINT constant to the endpoint to your AI Platform Pipelines instance. The endpoint to the AI Platform Pipelines instance can be found on the AI Platform Pipelines page in the Google Cloud Console. Open the SETTINGS for your instance and use the value of the host variable in the Connect to this Kubeflow Pipelines instance from a Python client via Kubeflow Pipelines SKD section of the SETTINGS window. The format is '...pipelines.googleusercontent.com'.
End of explanation
PIPELINE_NAME = 'tfx_covertype_continuous_training'
MODEL_NAME = 'tfx_covertype_classifier'
DATA_ROOT_URI = 'gs://workshop-datasets/covertype/small'
CUSTOM_TFX_IMAGE = 'gcr.io/{}/{}'.format(PROJECT_ID, PIPELINE_NAME)
RUNTIME_VERSION = '2.3'
PYTHON_VERSION = '3.7'
USE_KFP_SA=False
ENABLE_TUNING=False
%env PIPELINE_NAME={PIPELINE_NAME}
%env MODEL_NAME={MODEL_NAME}
%env DATA_ROOT_URI={DATA_ROOT_URI}
%env KUBEFLOW_TFX_IMAGE={CUSTOM_TFX_IMAGE}
%env RUNTIME_VERSION={RUNTIME_VERSION}
%env PYTHON_VERIONS={PYTHON_VERSION}
%env USE_KFP_SA={USE_KFP_SA}
%env ENABLE_TUNING={ENABLE_TUNING}
Explanation: Set the pipeline compile time settings
Default pipeline runtime environment values are configured in the pipeline folder config.py. You will set their values directly below:
PIPELINE_NAME - the pipeline's globally unique name. For each subsequent pipeline update, each pipeline version uploaded to KFP will be reflected on the Pipelines tab in the Pipeline name &gt; Version name dropdown in the format PIPELINE_NAME_datetime.now().
MODEL_NAME - the pipeline's unique model output name for AI Platform Prediction. For multiple pipeline runs, each pushed blessed model will create a new version with the format 'v{}'.format(int(time.time())).
DATA_ROOT_URI - the URI for the raw lab dataset gs://workshop-datasets/covertype/small.
CUSTOM_TFX_IMAGE - the image name of your pipeline container build by skaffold and published by Cloud Build to Cloud Container Registry in the format 'gcr.io/{}/{}'.format(PROJECT_ID, PIPELINE_NAME).
RUNTIME_VERSION - the TensorFlow runtime version. This lab was built and tested using TensorFlow 2.3.
PYTHON_VERSION - the Python runtime version. This lab was built and tested using Python 3.7.
USE_KFP_SA - The pipeline can run using a security context of the GKE default node pool's service account or the service account defined in the user-gcp-sa secret of the Kubernetes namespace hosting Kubeflow Pipelines. If you want to use the user-gcp-sa service account you change the value of USE_KFP_SA to True. Note that the default AI Platform Pipelines configuration does not define the user-gcp-sa secret.
ENABLE_TUNING - boolean value indicating whether to add the Tuner component to the pipeline or use hyperparameter defaults. See the model.py and pipeline.py files for details on how this changes the pipeline topology across pipeline versions.
End of explanation
!tfx pipeline compile --engine kubeflow --pipeline_path runner.py
Explanation: Compile your pipeline code
You can build and upload the pipeline to the AI Platform Pipelines instance in one step, using the tfx pipeline create command. The tfx pipeline create goes through the following steps:
- (Optional) Builds the custom image to that provides a runtime environment for TFX components or uses the latest image of the installed TFX version. 
- Compiles the pipeline code into a pipeline package. 
- Uploads the pipeline package via the ENDPOINT to the hosted AI Platform instance.
When prototyping with the TFX SDK, you may prefer to first use the tfx pipeline compile command, which only executes the compilation step. After the pipeline compiles successfully you can use tfx pipeline create to go through all steps.
End of explanation
!tfx pipeline create  \
--pipeline_path=runner.py \
--endpoint={ENDPOINT} \
--build_target_image={CUSTOM_TFX_IMAGE}
Explanation: Note: you should see a {PIPELINE_NAME}.tar.gz file appear in your /pipeline directory.
Exercise: deploy your pipeline container to AI Platform Pipelines with TFX CLI
After the pipeline code compiles without any errors you can use the tfx pipeline create command to perform the full build and deploy the pipeline. You will deploy your compiled pipeline container hosted on Google Container Registry e.g. gcr.io/[PROJECT_ID]/[PIPELINE_NAME] to run on AI Platform Pipelines with the TFX CLI. To learn more about the command below, you can review the TFX CLI documentation.
End of explanation
!tfx run create --pipeline_name={PIPELINE_NAME} --endpoint={ENDPOINT}
Explanation: If you make a mistake above and need to redeploy the pipeline you can first delete the previous version using tfx pipeline delete or you can update the pipeline in-place using tfx pipeline update.
To delete the pipeline:
tfx pipeline delete --pipeline_name {PIPELINE_NAME} --endpoint {ENDPOINT}
To update the pipeline:
tfx pipeline update --pipeline_path runner.py --endpoint {ENDPOINT}
Exercise: create a pipeline run with the TFX CLI
After the pipeline has been deployed, you can trigger and monitor pipeline runs using TFX CLI. For more information on the step below, review the TFX CLI documentation on the "run group".
End of explanation
!tfx run list --pipeline_name {PIPELINE_NAME} --endpoint {ENDPOINT}
Explanation: Exercise: monitor your pipeline runs with the TFX CLI
To view the status of existing pipeline runs:
End of explanation
RUN_ID='[YOUR RUN ID]'
!tfx run status --pipeline_name {PIPELINE_NAME} --run_id {RUN_ID} --endpoint {ENDPOINT}
Explanation: To retrieve the status of a given run retrieved from the command above:
End of explanation
<END_TASK>