<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 Github
Python2에서 진행
Python3에서도 되긴 하는데, 몇 기능이 안될듯(Apache Beam이 아직 파이썬2만 지원)
Step1:  Compute and visualize statistics
tfdv.generate_statistics_from_csv로 데이터 분포 생성
많은 데이터일 경우 내부적으로 Apache Beam을 사용해 병렬처리
Beam의 PTransform과 결합 가능
Step2:  tfdv.visualize_statistics를 사용해 시각화, 내부적으론 Facets을 사용한다 함
numeric, categorical feature들을 나눔
Step3:  Infer a scahema
데이터를 통해 스키마 추론
tfdv.infer_schema
tfdv.display_schema
Step4:  평가 데이터 에러 체크
train, validation에서 다른 데이터들이 있음
캐글할 때 유용할듯
Step5:  Check for evaluation anomalies
train 데이터엔 없었는데 validation에 생긴 데이터 있는지 확인
Step6:  Fix evaluation anomalies in the schema
수정
Step7:  Schema Environments
serving할 때도 스키마 체크해야 함
Environments can be used to express such requirements. In particular, features in schema can be associated with a set of environments using default_environment, in_environment and not_in_environment.
Step8:  Int value가 있음 => Float으로 수정
Step9:  Check for drift and skew
Drift
Drift detection is supported for categorical features and between consecutive spans of data (i.e., between span N and span N+1), such as between different days of training data. We express drift in terms of L-infinity distance, and you can set the threshold distance so that you receive warnings when the drift is higher than is acceptable. Setting the correct distance is typically an iterative process requiring domain knowledge and experimentation.
Skew
Schema Skew
같은 스키마를 가지지 않을 때
Feature Skew
Feature 생성 로직이 변경될 때
Dsitribution Skew
Train, Serving시 데이터 분포가 다를 경우
Step10:  Freeze the schema
스키마 저장
<ASSISTANT_TASK:>
Python Code:
from __future__ import print_function
import sys, os
import tempfile, urllib, zipfile
# Confirm that we're using Python 2
assert sys.version_info.major is 2, 'Oops, not running Python 2'
# Set up some globals for our file paths
BASE_DIR = tempfile.mkdtemp()
DATA_DIR = os.path.join(BASE_DIR, 'data')
OUTPUT_DIR = os.path.join(BASE_DIR, 'chicago_taxi_output')
TRAIN_DATA = os.path.join(DATA_DIR, 'train', 'data.csv')
EVAL_DATA = os.path.join(DATA_DIR, 'eval', 'data.csv')
SERVING_DATA = os.path.join(DATA_DIR, 'serving', 'data.csv')
# Download the zip file from GCP and unzip it
zip, headers = urllib.urlretrieve('https://storage.googleapis.com/tfx-colab-datasets/chicago_data.zip')
zipfile.ZipFile(zip).extractall(BASE_DIR)
zipfile.ZipFile(zip).close()
print("Here's what we downloaded:")
!ls -lR {os.path.join(BASE_DIR, 'data')}
!pip2 install -q tensorflow_data_validation
import tensorflow_data_validation as tfdv
print('TFDV version: {}'.format(tfdv.version.__version__))
Explanation: Github
Python2에서 진행
Python3에서도 되긴 하는데, 몇 기능이 안될듯(Apache Beam이 아직 파이썬2만 지원)
End of explanation
train_stats = tfdv.generate_statistics_from_csv(data_location=TRAIN_DATA)
Explanation: Compute and visualize statistics
tfdv.generate_statistics_from_csv로 데이터 분포 생성
많은 데이터일 경우 내부적으로 Apache Beam을 사용해 병렬처리
Beam의 PTransform과 결합 가능
End of explanation
tfdv.visualize_statistics(train_stats)
Explanation: tfdv.visualize_statistics를 사용해 시각화, 내부적으론 Facets을 사용한다 함
numeric, categorical feature들을 나눔
End of explanation
schema = tfdv.infer_schema(statistics=train_stats)
tfdv.display_schema(schema=schema)
Explanation: Infer a scahema
데이터를 통해 스키마 추론
tfdv.infer_schema
tfdv.display_schema
End of explanation
# Compute stats for evaluation data
eval_stats = tfdv.generate_statistics_from_csv(data_location=EVAL_DATA)
# Compare evaluation data with training data
tfdv.visualize_statistics(lhs_statistics=eval_stats, rhs_statistics=train_stats,
                          lhs_name='EVAL_DATASET', rhs_name='TRAIN_DATASET')
Explanation: 평가 데이터 에러 체크
train, validation에서 다른 데이터들이 있음
캐글할 때 유용할듯
End of explanation
# Check eval data for errors by validating the eval data stats using the previously inferred schema.
anomalies = tfdv.validate_statistics(statistics=eval_stats, schema=schema)
tfdv.display_anomalies(anomalies)
Explanation: Check for evaluation anomalies
train 데이터엔 없었는데 validation에 생긴 데이터 있는지 확인
End of explanation
# Relax the minimum fraction of values that must come from the domain for feature company.
company = tfdv.get_feature(schema, 'company')
company.distribution_constraints.min_domain_mass = 0.9
# Add new value to the domain of feature payment_type.
payment_type_domain = tfdv.get_domain(schema, 'payment_type')
payment_type_domain.value.append('Prcard')
# Validate eval stats after updating the schema 
updated_anomalies = tfdv.validate_statistics(eval_stats, schema)
tfdv.display_anomalies(updated_anomalies)
Explanation: Fix evaluation anomalies in the schema
수정
End of explanation
serving_stats = tfdv.generate_statistics_from_csv(SERVING_DATA)
serving_anomalies = tfdv.validate_statistics(serving_stats, schema)
tfdv.display_anomalies(serving_anomalies)
Explanation: Schema Environments
serving할 때도 스키마 체크해야 함
Environments can be used to express such requirements. In particular, features in schema can be associated with a set of environments using default_environment, in_environment and not_in_environment.
End of explanation
options = tfdv.StatsOptions(schema=schema, infer_type_from_schema=True)
serving_stats = tfdv.generate_statistics_from_csv(SERVING_DATA, stats_options=options)
serving_anomalies = tfdv.validate_statistics(serving_stats, schema)
tfdv.display_anomalies(serving_anomalies)
# All features are by default in both TRAINING and SERVING environments.
schema.default_environment.append('TRAINING')
schema.default_environment.append('SERVING')
# Specify that 'tips' feature is not in SERVING environment.
tfdv.get_feature(schema, 'tips').not_in_environment.append('SERVING')
serving_anomalies_with_env = tfdv.validate_statistics(
    serving_stats, schema, environment='SERVING')
tfdv.display_anomalies(serving_anomalies_with_env)
Explanation: Int value가 있음 => Float으로 수정
End of explanation
# Add skew comparator for 'payment_type' feature.
payment_type = tfdv.get_feature(schema, 'payment_type')
payment_type.skew_comparator.infinity_norm.threshold = 0.01
# Add drift comparator for 'company' feature.
company=tfdv.get_feature(schema, 'company')
company.drift_comparator.infinity_norm.threshold = 0.001
skew_anomalies = tfdv.validate_statistics(train_stats, schema,
                                          previous_statistics=eval_stats,
                                          serving_statistics=serving_stats)
tfdv.display_anomalies(skew_anomalies)
Explanation: Check for drift and skew
Drift
Drift detection is supported for categorical features and between consecutive spans of data (i.e., between span N and span N+1), such as between different days of training data. We express drift in terms of L-infinity distance, and you can set the threshold distance so that you receive warnings when the drift is higher than is acceptable. Setting the correct distance is typically an iterative process requiring domain knowledge and experimentation.
Skew
Schema Skew
같은 스키마를 가지지 않을 때
Feature Skew
Feature 생성 로직이 변경될 때
Dsitribution Skew
Train, Serving시 데이터 분포가 다를 경우
End of explanation
from tensorflow.python.lib.io import file_io
from google.protobuf import text_format
file_io.recursive_create_dir(OUTPUT_DIR)
schema_file = os.path.join(OUTPUT_DIR, 'schema.pbtxt')
tfdv.write_schema_text(schema, schema_file)
!cat {schema_file}
Explanation: Freeze the schema
스키마 저장
End of explanation
<END_TASK>