<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 Continuous Evaluation
This notebook demonstrates how to use Cloud AI Platform to execute continuous evaluation of a deployed machine learning model. You'll need to have a project set up with Google Cloud Platform. 
Set up
Start by creating environment variables for your Google Cloud project and bucket. Also, import the libraries we'll need for this notebook.
Step1:  Train and deploy the model
For this notebook, we'll build a text classification model using the Hacker News dataset. Each training example consists of an article title and the article source. The model will be trained to classify a given article title as belonging to either nytimes, github or techcrunch.
Load the data
Step2:  We one-hot encode the label...
Step3:  ...and create a train/test split.
Step4:  Swivel Model
We'll build a simple text classification model using a Tensorflow Hub embedding module derived from Swivel. Swivel is an algorithm that essentially factorizes word co-occurrence matrices to create the words embeddings. 
TF-Hub hosts the pretrained gnews-swivel-20dim-with-oov 20-dimensional Swivel module.
Step5:  The build_model function is written so that the TF Hub module can easily be exchanged with another module.
Step6:  Train and evaluation the model
With the model defined and data set up, next we'll train and evaluate the model.
Step7:  For training we'll call train_and_evaluate on txtcls_model.
Step8:  Calling predicition from model head produces output from final dense layer. This final layer is used to compute categorical cross-entropy when training.
Step9:  We can save the model artifacts in the local directory called ./txtcls_swivel.
Step10:  ....and examine the model's serving default signature. As expected the model takes as input a text string (e.g. an article title) and retrns a 3-dimensional vector of floats (i.e. the softmax output layer).
Step11:  To simplify the returned predictions, we'll modify the model signature so that the model outputs the predicted article source (either nytimes, techcrunch, or github) rather than the final softmax layer. We'll also return the 'confidence' of the model's prediction. This will be the softmax value corresonding to the predicted article source.
Step12:  Now, we'll re-save the new Swivel model that has this updated model signature by referencing the source_name function for the model's serving_default.
Step13:  Examine the model signature to confirm the changes
Step14:  Now when we call predictions using the updated serving input function, the model will return the predicted article source as a readable string, and the model's confidence for that prediction.
Step15:  Deploy the model for online serving
Once the model is trained and the assets saved, deploying the model to GCP is straightforward. After some time you should be able to see your deployed model and its version on the model page of GCP console.
Step16:  Set up the Evaluation job on CAIP
Now that the model is deployed, go to Cloud AI Platform to see the model version you've deployed and set up an evaluation job by clicking on the button called "Create Evaluation Job". You will be asked to provide some relevant information
Step17:  Now, every time this model version receives an online prediction request, this information will be captured and stored in the BQ table. Note, this happens everytime because we set the sampling proportion to 100%. 
Send prediction requests to your model
Here are some article titles and their groundtruth sources that we can test with prediciton.
| title  | groundtruth  |
|---|---|
| YouTube introduces Video Chapters to make it easier to navigate longer videos  |  techcrunch |
| A Filmmaker Put Away for Tax Fraud Takes Us Inside a British Prison | nytimes |
| A native Mac app wrapper for WhatsApp Web | github |
| Astronauts Dock With Space Station After Historic SpaceX Launch | nytimes |
| House Passes Sweeping Policing Bill Targeting Racial Bias and Use of Force | nytimes |
| Scrollability | github |
| iOS 14 lets deaf users set alerts for important sounds, among other clever accessibility perks | techcrunch |
Step18:  Summarizing the results from our model
Step19:  Provide the ground truth for the raw prediction input
Notice the groundtruth is missing. We'll update the evaluation table to contain the ground truth.
Step20:  We can confirm that the ground truch has been properly added to the table.
Step21:  Compute evaluation metrics
With the raw prediction input, the model output and the groundtruth in one place, we can evaluation how our model performs. And how the model performs across various aspects (e.g. over time, different model versions, different labels, etc)
Step22:  Using regex we can extract the model predictions, to have an easier to read format
Step23:  Or a full classification report from the sklearn library
Step24:  Can also examine a confusion matrix
Step25:  Examine eval metrics by model version or timestamp
By specifying the same evaluation table, two different model versions can be evaluated. Also, since the timestamp is captured, it is straightforward to evaluation model performance over time.
<ASSISTANT_TASK:>
Python Code:
# change these to try this notebook out
PROJECT = 'munn-sandbox'
BUCKET = 'munn-sandbox'
import os
os.environ['BUCKET'] = BUCKET
os.environ['PROJECT'] = PROJECT
os.environ['TFVERSION'] = '2.1'
import shutil
import pandas as pd
import tensorflow as tf
from google.cloud import bigquery
from tensorflow.keras.utils import to_categorical
from tensorflow.keras.callbacks import EarlyStopping
from tensorflow_hub import KerasLayer
from tensorflow.keras.layers import Dense, Input, Lambda
from tensorflow.keras.models import Model
print(tf.__version__)
%matplotlib inline
Explanation: Continuous Evaluation
This notebook demonstrates how to use Cloud AI Platform to execute continuous evaluation of a deployed machine learning model. You'll need to have a project set up with Google Cloud Platform. 
Set up
Start by creating environment variables for your Google Cloud project and bucket. Also, import the libraries we'll need for this notebook.
End of explanation
DATASET_NAME = "titles_full.csv"
COLUMNS = ['title', 'source']
titles_df = pd.read_csv(DATASET_NAME, header=None, names=COLUMNS)
titles_df.head()
Explanation: Train and deploy the model
For this notebook, we'll build a text classification model using the Hacker News dataset. Each training example consists of an article title and the article source. The model will be trained to classify a given article title as belonging to either nytimes, github or techcrunch.
Load the data
End of explanation
CLASSES = {
    'github': 0,
    'nytimes': 1,
    'techcrunch': 2
}
N_CLASSES = len(CLASSES)
def encode_labels(sources):
    classes = [CLASSES[source] for source in sources]
    one_hots = to_categorical(classes, num_classes=N_CLASSES)
    return one_hots
encode_labels(titles_df.source[:4])
Explanation: We one-hot encode the label...
End of explanation
N_TRAIN = int(len(titles_df) * 0.80)
titles_train, sources_train = (
    titles_df.title[:N_TRAIN], titles_df.source[:N_TRAIN])
titles_valid, sources_valid = (
    titles_df.title[N_TRAIN:], titles_df.source[N_TRAIN:])
X_train, Y_train = titles_train.values, encode_labels(sources_train)
X_valid, Y_valid = titles_valid.values, encode_labels(sources_valid)
X_train[:3]
Explanation: ...and create a train/test split.
End of explanation
SWIVEL = "https://tfhub.dev/google/tf2-preview/gnews-swivel-20dim-with-oov/1"
swivel_module = KerasLayer(SWIVEL, output_shape=[20], input_shape=[], dtype=tf.string, trainable=True)
Explanation: Swivel Model
We'll build a simple text classification model using a Tensorflow Hub embedding module derived from Swivel. Swivel is an algorithm that essentially factorizes word co-occurrence matrices to create the words embeddings. 
TF-Hub hosts the pretrained gnews-swivel-20dim-with-oov 20-dimensional Swivel module.
End of explanation
def build_model(hub_module, model_name):
    inputs = Input(shape=[], dtype=tf.string, name="text")
    module = hub_module(inputs)
    h1 = Dense(16, activation='relu', name="h1")(module)
    outputs = Dense(N_CLASSES, activation='softmax', name='outputs')(h1)
    model = Model(inputs=inputs, outputs=[outputs], name=model_name)
    
    model.compile(
        optimizer='adam',
        loss='categorical_crossentropy',
        metrics=['accuracy']
    )
    return model
def train_and_evaluate(train_data, val_data, model, batch_size=5000):
    tf.random.set_seed(33)    
    X_train, Y_train = train_data
    history = model.fit(
        X_train, Y_train,
        epochs=100,
        batch_size=batch_size,
        validation_data=val_data,
        callbacks=[EarlyStopping()],
    )
    return history
txtcls_model = build_model(swivel_module, model_name='txtcls_swivel')
txtcls_model.summary()
Explanation: The build_model function is written so that the TF Hub module can easily be exchanged with another module.
End of explanation
# set up train and validation data
train_data = (X_train, Y_train)
val_data = (X_valid, Y_valid)
Explanation: Train and evaluation the model
With the model defined and data set up, next we'll train and evaluate the model.
End of explanation
txtcls_history = train_and_evaluate(train_data, val_data, txtcls_model)
history = txtcls_history
pd.DataFrame(history.history)[['loss', 'val_loss']].plot()
pd.DataFrame(history.history)[['accuracy', 'val_accuracy']].plot()
Explanation: For training we'll call train_and_evaluate on txtcls_model.
End of explanation
txtcls_model.predict(x=["YouTube introduces Video Chapters to make it easier to navigate longer videos"])
Explanation: Calling predicition from model head produces output from final dense layer. This final layer is used to compute categorical cross-entropy when training.
End of explanation
tf.saved_model.save(txtcls_model, './txtcls_swivel/')
Explanation: We can save the model artifacts in the local directory called ./txtcls_swivel.
End of explanation
!saved_model_cli show \
 --tag_set serve \
 --signature_def serving_default \
 --dir ./txtcls_swivel/
Explanation: ....and examine the model's serving default signature. As expected the model takes as input a text string (e.g. an article title) and retrns a 3-dimensional vector of floats (i.e. the softmax output layer).
End of explanation
@tf.function(input_signature=[tf.TensorSpec([None], dtype=tf.string)])
def source_name(text):
    labels = tf.constant(['github', 'nytimes', 'techcrunch'], dtype=tf.string)
    probs = txtcls_model(text, training=False)
    indices = tf.argmax(probs, axis=1)
    pred_source = tf.gather(params=labels, indices=indices)
    pred_confidence = tf.reduce_max(probs, axis=1)
    
    return {'source': pred_source,
            'confidence': pred_confidence}
Explanation: To simplify the returned predictions, we'll modify the model signature so that the model outputs the predicted article source (either nytimes, techcrunch, or github) rather than the final softmax layer. We'll also return the 'confidence' of the model's prediction. This will be the softmax value corresonding to the predicted article source.
End of explanation
shutil.rmtree('./txtcls_swivel', ignore_errors=True)
txtcls_model.save('./txtcls_swivel', signatures={'serving_default': source_name})
Explanation: Now, we'll re-save the new Swivel model that has this updated model signature by referencing the source_name function for the model's serving_default.
End of explanation
!saved_model_cli show \
 --tag_set serve \
 --signature_def serving_default \
 --dir ./txtcls_swivel/
Explanation: Examine the model signature to confirm the changes:
End of explanation
title1 = "House Passes Sweeping Policing Bill Targeting Racial Bias and Use of Force"
title2 = "YouTube introduces Video Chapters to make it easier to navigate longer videos"
title3 = "As facebook turns 10  zuckerberg wants to change how tech industry works"
restored = tf.keras.models.load_model('./txtcls_swivel')
infer = restored.signatures['serving_default']
outputs = infer(text=tf.constant([title1, title2, title3]))
print(outputs['source'].numpy())
print(outputs['confidence'].numpy())
Explanation: Now when we call predictions using the updated serving input function, the model will return the predicted article source as a readable string, and the model's confidence for that prediction.
End of explanation
%%bash
MODEL_NAME="txtcls"
MODEL_VERSION="swivel"
MODEL_LOCATION="./txtcls_swivel/"
gcloud ai-platform models create ${MODEL_NAME}
gcloud ai-platform versions create ${MODEL_VERSION} \
--model ${MODEL_NAME} \
--origin ${MODEL_LOCATION} \
--staging-bucket gs://${BUCKET} \
--runtime-version=2.1
Explanation: Deploy the model for online serving
Once the model is trained and the assets saved, deploying the model to GCP is straightforward. After some time you should be able to see your deployed model and its version on the model page of GCP console.
End of explanation
%load_ext google.cloud.bigquery
%%bigquery --project $PROJECT
SELECT * FROM `txtcls_eval.swivel`
Explanation: Set up the Evaluation job on CAIP
Now that the model is deployed, go to Cloud AI Platform to see the model version you've deployed and set up an evaluation job by clicking on the button called "Create Evaluation Job". You will be asked to provide some relevant information:
 - Job description: txtcls_swivel_eval
 - Model objective: text classification
 - Classification type: single-label classification
 - Prediction label file path for the annotation specification set: When you create an evaluation job on CAIP, you must specify a CSV file that defines your annotation specification set. This file must have one row for every possible label your model outputs during prediction. Each row should be a comma-separated pair containing the label and a description of the label: label-name,description
 - Daily sample percentage: We'll set this to 100% so that all online predicitons are captured for evaluation.
 - BigQuery table to house online prediction requests: We'll use the BQ dataset and table txtcls_eval.swivel. If you enter a BigQuery table that doesn’t exist, one with that name will be created with the correct schema. 
 - Prediction input
     - Data key: this is The key for the raw prediction data. From examining our deployed model signature, the input data key is text.
     - Data reference key: this is for image models, so we can ignore
 - Prediction output
     - Prediction labels key: This is the prediction key which contains the predicted label (i.e. the article source). For our model, the label key is source.
     - Prediction score key: This is the prediction key which contains the predicted scores (i.e. the model confidence). For our model, the score key is confidence.
 - Ground-truth method: Check the box that indicates we will provide our own labels, and not use a Human data labeling service.
Once the evaluation job is set up, the table will be made in BigQuery to capture the online prediction requests.
End of explanation
%%writefile input.json
{"text": "YouTube introduces Video Chapters to make it easier to navigate longer videos"}
!gcloud ai-platform predict \
  --model txtcls \
  --json-instances input.json \
  --version swivel
%%writefile input.json
{"text": "A Filmmaker Put Away for Tax Fraud Takes Us Inside a British Prison"}
!gcloud ai-platform predict \
  --model txtcls \
  --json-instances input.json \
  --version swivel
%%writefile input.json
{"text": "A native Mac app wrapper for WhatsApp Web"}
!gcloud ai-platform predict \
  --model txtcls \
  --json-instances input.json \
  --version swivel
%%writefile input.json
{"text": "Astronauts Dock With Space Station After Historic SpaceX Launch"}
!gcloud ai-platform predict \
  --model txtcls \
  --json-instances input.json \
  --version swivel
%%writefile input.json
{"text": "House Passes Sweeping Policing Bill Targeting Racial Bias and Use of Force"}
!gcloud ai-platform predict \
  --model txtcls \
  --json-instances input.json \
  --version swivel
%%writefile input.json
{"text": "Scrollability"}
!gcloud ai-platform predict \
  --model txtcls \
  --json-instances input.json \
  --version swivel
%%writefile input.json
{"text": "iOS 14 lets deaf users set alerts for important sounds, among other clever accessibility perks"}
!gcloud ai-platform predict \
  --model txtcls \
  --json-instances input.json \
  --version swivel
Explanation: Now, every time this model version receives an online prediction request, this information will be captured and stored in the BQ table. Note, this happens everytime because we set the sampling proportion to 100%. 
Send prediction requests to your model
Here are some article titles and their groundtruth sources that we can test with prediciton.
| title  | groundtruth  |
|---|---|
| YouTube introduces Video Chapters to make it easier to navigate longer videos  |  techcrunch |
| A Filmmaker Put Away for Tax Fraud Takes Us Inside a British Prison | nytimes |
| A native Mac app wrapper for WhatsApp Web | github |
| Astronauts Dock With Space Station After Historic SpaceX Launch | nytimes |
| House Passes Sweeping Policing Bill Targeting Racial Bias and Use of Force | nytimes |
| Scrollability | github |
| iOS 14 lets deaf users set alerts for important sounds, among other clever accessibility perks | techcrunch |
End of explanation
%%bigquery --project $PROJECT
SELECT * FROM `txtcls_eval.swivel`
Explanation: Summarizing the results from our model:
| title  | groundtruth  | predicted
|---|---|---|
| YouTube introduces Video Chapters to make it easier to navigate longer videos  |  techcrunch | techcrunch |
| A Filmmaker Put Away for Tax Fraud Takes Us Inside a British Prison | nytimes | techcrunch |
| A native Mac app wrapper for WhatsApp Web | github | techcrunch |
| Astronauts Dock With Space Station After Historic SpaceX Launch | nytimes | techcrunch |
| House Passes Sweeping Policing Bill Targeting Racial Bias and Use of Force | nytimes | nytimes |
| Scrollability | github | techcrunch |
| iOS 14 lets deaf users set alerts for important sounds, among other clever accessibility perks | techcrunch | nytimes |
End of explanation
%%bigquery --project $PROJECT
UPDATE `txtcls_eval.swivel`
SET 
    groundtruth = '{"predictions": [{"source": "techcrunch"}]}'
WHERE
    raw_data = '{"instances": [{"text": "YouTube introduces Video Chapters to make it easier to navigate longer videos"}]}';
%%bigquery --project $PROJECT
UPDATE `txtcls_eval.swivel`
SET 
    groundtruth = '{"predictions": [{"source": "nytimes"}]}'
WHERE
    raw_data = '{"instances": [{"text": "A Filmmaker Put Away for Tax Fraud Takes Us Inside a British Prison"}]}';
%%bigquery --project $PROJECT
UPDATE `txtcls_eval.swivel`
SET 
    groundtruth = '{"predictions": [{"source": "github"}]}'
WHERE
    raw_data = '{"instances": [{"text": "A native Mac app wrapper for WhatsApp Web"}]}';
%%bigquery --project $PROJECT
UPDATE `txtcls_eval.swivel`
SET 
    groundtruth = '{"predictions": [{"source": "nytimes"}]}'
WHERE
    raw_data = '{"instances": [{"text": "Astronauts Dock With Space Station After Historic SpaceX Launch"}]}';
%%bigquery --project $PROJECT
UPDATE `txtcls_eval.swivel`
SET 
    groundtruth = '{"predictions": [{"source": "nytimes"}]}'
WHERE
    raw_data = '{"instances": [{"text": "House Passes Sweeping Policing Bill Targeting Racial Bias and Use of Force"}]}';
%%bigquery --project $PROJECT
UPDATE `txtcls_eval.swivel`
SET 
    groundtruth = '{"predictions": [{"source": "github"}]}'
WHERE
    raw_data = '{"instances": [{"text": "Scrollability"}]}';
%%bigquery --project $PROJECT
UPDATE `txtcls_eval.swivel`
SET 
    groundtruth = '{"predictions": [{"source": "techcrunch"}]}'
WHERE
    raw_data = '{"instances": [{"text": "iOS 14 lets deaf users set alerts for important sounds, among other clever accessibility perks"}]}';
Explanation: Provide the ground truth for the raw prediction input
Notice the groundtruth is missing. We'll update the evaluation table to contain the ground truth.
End of explanation
%%bigquery --project $PROJECT
SELECT * FROM `txtcls_eval.swivel`
Explanation: We can confirm that the ground truch has been properly added to the table.
End of explanation
import seaborn as sns
import pandas as pd
import matplotlib.pyplot as plt
from sklearn.metrics import confusion_matrix
from sklearn.metrics import precision_recall_fscore_support as score
from sklearn.metrics import classification_report
Explanation: Compute evaluation metrics
With the raw prediction input, the model output and the groundtruth in one place, we can evaluation how our model performs. And how the model performs across various aspects (e.g. over time, different model versions, different labels, etc)
End of explanation
%%bigquery --project $PROJECT
SELECT
  model,
  model_version,
  time,
  REGEXP_EXTRACT(raw_data, r'.*"text": "(.*)"') AS text,
  REGEXP_EXTRACT(raw_prediction, r'.*"source": "(.*?)"') AS prediction,
  REGEXP_EXTRACT(raw_prediction, r'.*"confidence": (0.\d{2}).*') AS confidence,
  REGEXP_EXTRACT(groundtruth, r'.*"source": "(.*?)"') AS groundtruth,
FROM
  `txtcls_eval.swivel`
query = '''
SELECT
  model,
  model_version,
  time,
  REGEXP_EXTRACT(raw_data, r'.*"text": "(.*)"') AS text,
  REGEXP_EXTRACT(raw_prediction, r'.*"source": "(.*?)"') AS prediction,
  REGEXP_EXTRACT(raw_prediction, r'.*"confidence": (0.\d{2}).*') AS confidence,
  REGEXP_EXTRACT(groundtruth, r'.*"source": "(.*?)"') AS groundtruth,
FROM
  `txtcls_eval.swivel`
'''
client = bigquery.Client()
df_results = client.query(query).to_dataframe()
df_results.head(20)
prediction = list(df_results.prediction)
groundtruth = list(df_results.groundtruth)
precision, recall, fscore, support = score(groundtruth, prediction)
from tabulate import tabulate
sources = list(CLASSES.keys())
results = list(zip(sources, precision, recall, fscore, support))
print(tabulate(results, headers = ['source', 'precision', 'recall', 'fscore', 'support'],
         tablefmt='orgtbl'))
Explanation: Using regex we can extract the model predictions, to have an easier to read format:
End of explanation
print(classification_report(y_true=groundtruth, y_pred=prediction))
Explanation: Or a full classification report from the sklearn library:
End of explanation
cm = confusion_matrix(groundtruth, prediction, labels=sources)
ax= plt.subplot()
sns.heatmap(cm, annot=True, ax = ax, cmap="Blues")
# labels, title and ticks
ax.set_xlabel('Predicted labels')
ax.set_ylabel('True labels')
ax.set_title('Confusion Matrix') 
ax.xaxis.set_ticklabels(sources)
ax.yaxis.set_ticklabels(sources)
plt.savefig("./txtcls_cm.png")
Explanation: Can also examine a confusion matrix:
End of explanation
now = pd.Timestamp.now(tz='UTC')
one_week_ago = now - pd.DateOffset(weeks=1)
one_month_ago = now - pd.DateOffset(months=1)
df_prev_week = df_results[df_results.time > one_week_ago]
df_prev_month = df_results[df_results.time > one_month_ago]
df_prev_month
Explanation: Examine eval metrics by model version or timestamp
By specifying the same evaluation table, two different model versions can be evaluated. Also, since the timestamp is captured, it is straightforward to evaluation model performance over time.
End of explanation
<END_TASK>