<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 ML with TensorFlow Extended (TFX) -- Part 1
The puprpose of this tutorial is to show how to do end-to-end ML with TFX libraries on Google Cloud Platform. This tutorial covers
Step1:  <img valign="middle" src="images/tfx.jpeg">
UCI Adult Dataset
Step2:  1. Data Analysis
For data analysis, visualization, and schema generation, we use TensorFlow Data Validation to perform the following
Step3:  1.2 Infer Schema
Step4:  1.3 Configure Schema
Step5:  1.4 Validate evaluation data
Step6:  1.5 Freeze the schema
<ASSISTANT_TASK:>
Python Code:
import tensorflow as tf
import tensorflow_data_validation as tfdv
print('TF version: {}'.format(tf.__version__))
print('TFDV version: {}'.format(tfdv.__version__))
PROJECT = 'cloud-training-demos'    # Replace with your PROJECT
BUCKET = 'cloud-training-demos-ml'  # Replace with your BUCKET
REGION = 'us-central1'              # Choose an available region for Cloud MLE
import os
os.environ['PROJECT'] = PROJECT
os.environ['BUCKET'] = BUCKET
os.environ['REGION'] = REGION
%%bash
gcloud config set project $PROJECT
gcloud config set compute/region $REGION
## ensure we predict locally with our current Python environment
gcloud config set ml_engine/local_python `which python`
Explanation: ML with TensorFlow Extended (TFX) -- Part 1
The puprpose of this tutorial is to show how to do end-to-end ML with TFX libraries on Google Cloud Platform. This tutorial covers:
1. Data analysis and schema generation with TF Data Validation.
2. Data preprocessing with TF Transform.
3. Model training with TF Estimator.
4. Model evaluation with TF Model Analysis.
This notebook has been tested in Jupyter on the Deep Learning VM.
Setup Cloud environment
End of explanation
DATA_DIR='gs://cloud-samples-data/ml-engine/census/data'
import os
TRAIN_DATA_FILE = os.path.join(DATA_DIR, 'adult.data.csv')
EVAL_DATA_FILE = os.path.join(DATA_DIR, 'adult.test.csv')
!gsutil ls -l $TRAIN_DATA_FILE
!gsutil ls -l $EVAL_DATA_FILE
Explanation: <img valign="middle" src="images/tfx.jpeg">
UCI Adult Dataset: https://archive.ics.uci.edu/ml/datasets/adult
Predict whether income exceeds $50K/yr based on census data. Also known as "Census Income" dataset.
End of explanation
HEADER = ['age', 'workclass', 'fnlwgt', 'education', 'education_num',
               'marital_status', 'occupation', 'relationship', 'race', 'gender',
               'capital_gain', 'capital_loss', 'hours_per_week',
               'native_country', 'income_bracket']
TARGET_FEATURE_NAME = 'income_bracket'
TARGET_LABELS = [' <=50K', ' >50K']
WEIGHT_COLUMN_NAME = 'fnlwgt'
# This is a convenience function for CSV. We can write a Beam pipeline for other formats.
# https://www.tensorflow.org/tfx/data_validation/api_docs/python/tfdv/generate_statistics_from_csv
train_stats = tfdv.generate_statistics_from_csv(
    data_location=TRAIN_DATA_FILE, 
    column_names=HEADER,
    stats_options=tfdv.StatsOptions(
        weight_feature=WEIGHT_COLUMN_NAME,
        sample_rate=1.0
    )
)
tfdv.visualize_statistics(train_stats)
Explanation: 1. Data Analysis
For data analysis, visualization, and schema generation, we use TensorFlow Data Validation to perform the following:
1. Analyze the training data and produce statistics.
2. Generate data schema from the produced statistics.
3. Configure the schema.
4. Validate the evaluation data against the schema.
5. Save the schema for later use.
1.1 Compute and visualise statistics
End of explanation
schema = tfdv.infer_schema(statistics=train_stats)
tfdv.display_schema(schema=schema)
print(tfdv.get_feature(schema, 'age'))
Explanation: 1.2 Infer Schema
End of explanation
# Relax the minimum fraction of values that must come from the domain for feature occupation.
occupation = tfdv.get_feature(schema, 'occupation')
occupation.distribution_constraints.min_domain_mass = 0.9
# Add new value to the domain of feature native_country, assuming that we start receiving this
# we won't be able to make great predictions of course, because this country is not part of our
# training data.
native_country_domain = tfdv.get_domain(schema, 'native_country')
native_country_domain.value.append('Egypt')
# All features are by default in both TRAINING and SERVING environments.
schema.default_environment.append('TRAINING')
schema.default_environment.append('EVALUATION')
schema.default_environment.append('SERVING')
# Specify that the class feature is not in SERVING environment.
tfdv.get_feature(schema, TARGET_FEATURE_NAME).not_in_environment.append('SERVING')
tfdv.display_schema(schema=schema)
Explanation: 1.3 Configure Schema
End of explanation
eval_stats = tfdv.generate_statistics_from_csv(
    EVAL_DATA_FILE, 
    column_names=HEADER,
    stats_options=tfdv.StatsOptions(
        weight_feature=WEIGHT_COLUMN_NAME)
)
eval_anomalies = tfdv.validate_statistics(eval_stats, schema, environment='EVALUATION')
tfdv.display_anomalies(eval_anomalies)
Explanation: 1.4 Validate evaluation data
End of explanation
RAW_SCHEMA_LOCATION = 'raw_schema.pbtxt'
from tensorflow.python.lib.io import file_io
from google.protobuf import text_format
tfdv.write_schema_text(schema, RAW_SCHEMA_LOCATION)
!cat {RAW_SCHEMA_LOCATION}
Explanation: 1.5 Freeze the schema
End of explanation
<END_TASK>