<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 Building a Fraud Detection model on Vertex AI with TensorFlow Enterprise and BigQuery
Learning objectives
Analyze the data in BigQuery.
Ingest records from BigQuery.
Preprocess the data.
Build the model.
Train the model.
Evaluate the model.
Introduction
In this notebook, you'll directly ingest a BigQuery dataset and train a fraud detection model with TensorFlow Enterprise on Vertex AI.
You've also walked through all the steps of building a model. Finally, you learned a bit about how to handle imbalanced classification problems.
Each learning objective will correspond to a #TODO in the student lab notebook -- try to complete that notebook first before reviewing this solution notebook. 
Ingest records from BigQuery
Step 1
Step1:  Step 2
Step2:  Step 3
Step3:  Step 4
Step4:  Build the model
Step 1
Step5:  Step 2
Step6:  Step 3
Step7:  Step 4
<ASSISTANT_TASK:>
Python Code:
import tensorflow as tf
import tensorflow.keras as keras
import tensorflow.keras.layers as layers
from tensorflow_io.bigquery import BigQueryClient
import functools
Explanation: Building a Fraud Detection model on Vertex AI with TensorFlow Enterprise and BigQuery
Learning objectives
Analyze the data in BigQuery.
Ingest records from BigQuery.
Preprocess the data.
Build the model.
Train the model.
Evaluate the model.
Introduction
In this notebook, you'll directly ingest a BigQuery dataset and train a fraud detection model with TensorFlow Enterprise on Vertex AI.
You've also walked through all the steps of building a model. Finally, you learned a bit about how to handle imbalanced classification problems.
Each learning objective will correspond to a #TODO in the student lab notebook -- try to complete that notebook first before reviewing this solution notebook. 
Ingest records from BigQuery
Step 1: Import Python packages
Run the below cell to import the python packages.
End of explanation
GCP_PROJECT_ID = 'qwiklabs-gcp-00-b1e00ce17168' # Replace with your Project-ID
DATASET_GCP_PROJECT_ID = GCP_PROJECT_ID # A copy of the data is saved in the user project
DATASET_ID = 'tfe_codelab'
TRAIN_TABLE_ID = 'ulb_fraud_detection_train'
VAL_TABLE_ID = 'ulb_fraud_detection_val'
TEST_TABLE_ID = 'ulb_fraud_detection_test'
FEATURES = ['Time','V1','V2','V3','V4','V5','V6','V7','V8','V9','V10','V11','V12','V13','V14','V15','V16','V17','V18','V19','V20','V21','V22','V23','V24','V25','V26','V27','V28','Amount']
LABEL='Class'
DTYPES=[tf.float64] * len(FEATURES) + [tf.int64]
Explanation: Step 2: Define constants
Let's next define some constants for use in the project. Change GCP_PROJECT_ID to the actual project ID you are using. Go ahead and run new cells as you create them.
End of explanation
client = BigQueryClient()
def read_session(TABLE_ID):
    return client.read_session(
        "projects/" + GCP_PROJECT_ID, DATASET_GCP_PROJECT_ID, TABLE_ID, DATASET_ID,
        FEATURES + [LABEL], DTYPES, requested_streams=2
)
def extract_labels(input_dict):
  features = dict(input_dict)
  label = tf.cast(features.pop(LABEL), tf.float64)
  return (features, label)
Explanation: Step 3: Define helper functions
Now, let's define a couple functions. read_session() reads data from a BigQuery table. extract_labels() is a helper function to separate the label column from the rest, so that the dataset is in the format expected by keras.model_fit() later on.
End of explanation
BATCH_SIZE = 32
# TODO 1
# Create the datasets
raw_train_data = read_session(TRAIN_TABLE_ID).parallel_read_rows().map(extract_labels).batch(BATCH_SIZE)
raw_val_data = read_session(VAL_TABLE_ID).parallel_read_rows().map(extract_labels).batch(BATCH_SIZE)
raw_test_data = read_session(TEST_TABLE_ID).parallel_read_rows().map(extract_labels).batch(BATCH_SIZE)
next(iter(raw_train_data)) # Print first batch
Explanation: Step 4: Ingest data
Finally, let's create each dataset and then print the first batch from the training dataset. Note that we have defined a BATCH_SIZE of 32. This is an important parameter that will impact the speed and accuracy of training.
End of explanation
# TODO 2
MEANS = [94816.7387536405, 0.0011219465482001268, -0.0021445914636999603, -0.002317402958335562,
         -0.002525792169927835, -0.002136576923287782, -3.7586818983702984, 8.135919975738768E-4,
         -0.0015535579268265718, 0.001436137140461279, -0.0012193712736681508, -4.5364970422902533E-4,
         -4.6175444671576083E-4, 9.92177789685366E-4, 0.002366229151475428, 6.710217226762278E-4,
         0.0010325807119864225, 2.557260815835395E-4, -2.0804190062322664E-4, -5.057391100818653E-4,
         -3.452114767842334E-6, 1.0145936326270006E-4, 3.839214074518535E-4, 2.2061197469126577E-4,
         -1.5601580596677608E-4, -8.235017846415852E-4, -7.298316615408554E-4, -6.898459943652376E-5,
         4.724125688297753E-5, 88.73235686453587]
def norm_data(mean, data):
  data = tf.cast(data, tf.float32) * 1/(2*mean)
  return tf.reshape(data, [-1, 1])
numeric_columns = []
for i, feature in enumerate(FEATURES):
  num_col = tf.feature_column.numeric_column(feature, normalizer_fn=functools.partial(norm_data, MEANS[i]))
  numeric_columns.append(num_col)
numeric_columns
Explanation: Build the model
Step 1: Preprocess data
Let's create feature columns for each feature in the dataset. In this particular dataset, all of the columns are of type numeric_column, but there a number of other column types (e.g. categorical_column).
You will also norm the data to center around zero so that the network converges faster. You've precalculated the means of each feature to use in this calculation.
End of explanation
# TODO 3
model = keras.Sequential([
    tf.keras.layers.DenseFeatures(numeric_columns),
    layers.Dense(64, activation='relu'),
    layers.Dense(64, activation='relu'),
    layers.Dense(1, activation='sigmoid')
])
# Compile the model
model.compile(loss='binary_crossentropy',
              optimizer='adam',
              metrics=['accuracy', tf.keras.metrics.AUC(curve='PR')])
Explanation: Step 2: Build the model
Now we are ready to create a model. We will feed the columns we just created into the network. Then we will compile the model. We are including the Precision/Recall AUC metric, which is useful for imbalanced datasets.
End of explanation
# TODO 4
CLASS_WEIGHT = {
    0: 1,
    1: 100
}
EPOCHS = 3
train_data = raw_train_data.shuffle(10000)
val_data = raw_val_data
test_data = raw_test_data
# Train the model using model.fit()
model.fit(train_data, validation_data=val_data, class_weight=CLASS_WEIGHT, epochs=EPOCHS)
Explanation: Step 3: Train the model
There are a number of techniques to handle imbalanced data, including oversampling (generating new data in the minority class) and undersampling (reducing the data in the majority class).
For the purposes of this codelab, let's use a technique that overweights the loss when misclassifying the minority class. You'll specify a class_weight parameter when training and weight "1" (fraud) higher, since it is much less prevalent.
You will use 3 epochs (passes through the data) in this lab so training is quicker. In a real-world scenario, You'd want to run it long enough to the point where the stop seeing increases in accuracy of the validation set.
End of explanation
# TODO 5
# Evaluate the model
model.evaluate(test_data)
Explanation: Step 4: Evaluate the model
The evaluate() function can be applied to test data that the model has never seen to provide an objective assessment. Fortunately, we've set aside test data just for that!
End of explanation
<END_TASK>