<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 Introduction to TensorFlow Data Validation
Learning Objectives
Review TFDV methods
Generate statistics
Visualize statistics
Infer a schema
Update a schema
Introduction
This lab is an introduction to TensorFlow Data Validation (TFDV), a key component of TensorFlow Extended.  This lab serves as a foundation for understanding the features of TFDV and how it can help you understand, validate, and monitor your data. 
TFDV can be used for generating schemas and statistics about the distribution of every feature in the dataset. Such information is useful for comparing multiple datasets (e.g. training vs inference datasets) and reporting
Step1:  Restart the kernel (Kernel > Restart kernel > Restart).
Re-run the above cell and proceed further.
Note
Step2:  Load the Consumer Spending Dataset
We will download our dataset from Google Cloud Storage. The columns in the dataset are
Step3:  Review the methods present in TFDV
Step4:  Describing data with TFDV
The usual workflow when using TFDV during training is as follows
Step5:  Now let's use tfdv.visualize_statistics, which uses Facets to create a succinct visualization of our training data
Step6:  TFDV generates different types of statistics based on the type of features.
For numerical features, TFDV computes for every feature
Step7:  Infer a schema
Now let's use tfdv.infer_schema to create a schema for our data.  A schema defines constraints for the data that are relevant for ML. Example constraints include the data type of each feature, whether it's numerical or categorical, or the frequency of its presence in the data.  For categorical features the schema also defines the domain - the list of acceptable values.  Since writing a schema can be a tedious task, especially for datasets with lots of features, TFDV provides a method to generate an initial version of the schema based on the descriptive statistics.
Getting the schema right is important because the rest of our production pipeline will be relying on the schema that TFDV generates to be correct.  
Generating Schema
Once statistics are generated, the next step is to generate a schema for our dataset. This schema will map each feature in the dataset to a type (float, bytes, etc.). Also define feature boundaries (min, max, distribution of values and missings, etc.).
Link to infer schema
https
Step8:  The schema also provides documentation for the data, and so is useful when different developers work on the same data.  Let's use tfdv.display_schema to display the inferred schema so that we can review it.
Step9:  TFDV provides a API to print a summary of each feature schema using
In this visualization, the columns stand for
Step10:  Update a feature with a new value
Let's add "self-employed" to the Profession feature
Step11:  Let's remove "Homemaker" from "Profession"
Step12:  Change a feature from a float to an integer
<ASSISTANT_TASK:>
Python Code:
!pip install pyarrow==5.0.0
!pip install numpy==1.19.2
!pip install tensorflow-data-validation
Explanation: Introduction to TensorFlow Data Validation
Learning Objectives
Review TFDV methods
Generate statistics
Visualize statistics
Infer a schema
Update a schema
Introduction
This lab is an introduction to TensorFlow Data Validation (TFDV), a key component of TensorFlow Extended.  This lab serves as a foundation for understanding the features of TFDV and how it can help you understand, validate, and monitor your data. 
TFDV can be used for generating schemas and statistics about the distribution of every feature in the dataset. Such information is useful for comparing multiple datasets (e.g. training vs inference datasets) and reporting:
Statistical differences in the features distribution
TFDV also offers visualization capabilities for comparing datasets based on the Google PAIR Facets project.  
Each learning objective will correspond to a #TODO in this student lab notebook -- try to complete this notebook first and then review the Solution Notebook for reference. 
Import Libraries
End of explanation
import pandas as pd
import tensorflow_data_validation as tfdv
import sys
import warnings
warnings.filterwarnings('ignore')
print('Installing TensorFlow Data Validation')
!pip install -q tensorflow_data_validation[visualization]
print('TFDV version: {}'.format(tfdv.version.__version__))
# Confirm that we're using Python 3
assert sys.version_info.major is 3, 'Oops, not running Python 3. Use Runtime > Change runtime type'
Explanation: Restart the kernel (Kernel > Restart kernel > Restart).
Re-run the above cell and proceed further.
Note: Please ignore any incompatibility warnings and errors.
End of explanation
# Read a comma-separated values (csv) file into DataFrame.
# TODO: Your code goes here
score_train.head() 
# Read a comma-separated values (csv) file into DataFrame.
# TODO: Your code goes here
score_test.head()
score_train.info()
Explanation: Load the Consumer Spending Dataset
We will download our dataset from Google Cloud Storage. The columns in the dataset are:
'Graduated': Whether or not the person is a college graduate
'Work Experience': The number of years in the workforce
'Family Size': The size of the family unit
'Spending Score': The spending score for consumer spending
End of explanation
# check methods present in tfdv
# TODO: Your code goes here
Explanation: Review the methods present in TFDV
End of explanation
# Compute data statistics for the input pandas DataFrame.
# TODO: Your code goes here
Explanation: Describing data with TFDV
The usual workflow when using TFDV during training is as follows:
Generate statistics for the data
Use those statistics to generate a schema for each feature
Visualize the schema and statistics and manually inspect them
Update the schema if needed
Compute and visualize statistics
First we'll use tfdv.generate_statistics_from_csv to compute statistics for our training data. (ignore the snappy warnings)
TFDV can compute descriptive statistics that provide a quick overview of the data in terms of the features that are present and the shapes of their value distributions.
Internally, TFDV uses Apache Beam's data-parallel processing framework to scale the computation of statistics over large datasets. For applications that wish to integrate deeper with TFDV (e.g., attach statistics generation at the end of a data-generation pipeline), the API also exposes a Beam PTransform for statistics generation.
NOTE:  Compute statistics
* tfdv.generate_statistics_from_csv
* tfdv.generate_statistics_from_dataframe
* tfdv.generate_statistics_from_tfrecord
Generate Statistics from a Pandas DataFrame
End of explanation
# Visualize the input statistics using Facets.
# TODO: Your code goes here
Explanation: Now let's use tfdv.visualize_statistics, which uses Facets to create a succinct visualization of our training data:
Notice that numeric features and categorical features are visualized separately, and that charts are displayed showing the distributions for each feature.
Notice that features with missing or zero values display a percentage in red as a visual indicator that there may be issues with examples in those features.  The percentage is the percentage of examples that have missing or zero values for that feature.
Notice that there are no examples with values for pickup_census_tract.  This is an opportunity for dimensionality reduction!
Try clicking "expand" above the charts to change the display
Try hovering over bars in the charts to display bucket ranges and counts
Try switching between the log and linear scales, and notice how the log scale reveals much more detail about the payment_type categorical feature
Try selecting "quantiles" from the "Chart to show" menu, and hover over the markers to show the quantile percentages
End of explanation
train_stats = tfdv.generate_statistics_from_dataframe(dataframe=score_train)
test_stats = tfdv.generate_statistics_from_dataframe(dataframe=score_test)
tfdv.visualize_statistics(
  lhs_statistics=train_stats, lhs_name='TRAIN_DATASET',
  rhs_statistics=test_stats, rhs_name='NEW_DATASET')
Explanation: TFDV generates different types of statistics based on the type of features.
For numerical features, TFDV computes for every feature:
* Count of records
* Number of missing (i.e. null values)
* Histogram of values
* Mean and standard deviation
* Minimum and maximum values
* Percentage of zero values
For categorical features, TFDV provides:
* Count of values
* Percentage of missing values
* Number of unique values
* Average string length
* Count for each label and its rank
Let's compare the score_train and the score_test datasets
End of explanation
# Infers schema from the input statistics.
# TODO: Your code goes here
print(schema)
Explanation: Infer a schema
Now let's use tfdv.infer_schema to create a schema for our data.  A schema defines constraints for the data that are relevant for ML. Example constraints include the data type of each feature, whether it's numerical or categorical, or the frequency of its presence in the data.  For categorical features the schema also defines the domain - the list of acceptable values.  Since writing a schema can be a tedious task, especially for datasets with lots of features, TFDV provides a method to generate an initial version of the schema based on the descriptive statistics.
Getting the schema right is important because the rest of our production pipeline will be relying on the schema that TFDV generates to be correct.  
Generating Schema
Once statistics are generated, the next step is to generate a schema for our dataset. This schema will map each feature in the dataset to a type (float, bytes, etc.). Also define feature boundaries (min, max, distribution of values and missings, etc.).
Link to infer schema
https://www.tensorflow.org/tfx/data_validation/api_docs/python/tfdv/infer_schema
With TFDV, we generate schema from statistics using
End of explanation
tfdv.display_schema(schema=schema)
Explanation: The schema also provides documentation for the data, and so is useful when different developers work on the same data.  Let's use tfdv.display_schema to display the inferred schema so that we can review it.
End of explanation
# Update Family_Size from FLOAT to Int
Graduated_feature = tfdv.get_feature(schema, 'Graduated')
Graduated_feature.presence.min_fraction = 1.0
Profession_feature = tfdv.get_feature(schema, 'Profession')
Profession_feature.presence.min_fraction = 1.0
Family_Size_feature = tfdv.get_feature(schema, 'Family_Size')
Family_Size_feature.presence.min_fraction = 1.0
tfdv.display_schema(schema)
Explanation: TFDV provides a API to print a summary of each feature schema using
In this visualization, the columns stand for:
Type indicates the feature datatype.
Presence indicates whether the feature must be present in 100% of examples (required) or not (optional).
Valency indicates the number of values required per training example. 
Domain and Values indicates The feature domain and its values
In the case of categorical features, single indicates that each training example must have exactly one category for the feature.
Updating the Schema
As stated above, Presence indicates whether the feature must be present in 100% of examples (required) or not (optional).  Currently, all of our features except for our target label are shown as "optional". We need to make our features all required except for "Work Experience".  We will need to update the schema.
TFDV lets you update the schema according to your domain knowledge of the data if you are not satisfied by the auto-generated schema.  We will update three use cases:  Making a feature required, adding a value to a feature, and change a feature from a float to an integer. 
Change optional features to required.
End of explanation
Profession_domain = tfdv.get_domain(schema, 'Profession')
Profession_domain.value.insert(0, 'Self-Employed')
Profession_domain.value
# [0 indicates I want 'Self-Employed to come first', if the number were 3, 
# it would be placed after the third value. ]
Explanation: Update a feature with a new value
Let's add "self-employed" to the Profession feature
End of explanation
Profession_domain = tfdv.get_domain(schema, 'Profession')
Profession_domain.value.remove('Homemaker')
Profession_domain.value
Explanation: Let's remove "Homemaker" from "Profession"
End of explanation
# Update Family_Size to Int
size = tfdv.get_feature(schema, 'Family_Size')
size.type=2
tfdv.display_schema(schema)
Explanation: Change a feature from a float to an integer
End of explanation
<END_TASK>