<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 Copyright 2020 The TensorFlow Authors.
Step1:  FaceSSD Fairness Indicators Example Colab
<table class="tfo-notebook-buttons" align="left">
  <td>
    <a target="_blank" href="https
Step2:  Download and Understand the Data
Labeled Faces in the Wild is a public benchmark dataset for face verification, also known as pair matching. LFW contains more than 13,000 images of faces collected from the web.
We ran FaceSSD predictions on this dataset to predict whether a face is present in a given image. In this Colab, we will slice data according to gender to observe if there are any significant differences between model performance for different gender groups.
If there is more than one face in an image, gender is labeled as "MISSING".
We've hosted the dataset on Google Cloud Platform for convenience. Run the following code to download the data from GCP, the data will take about a minute to download and analyze.
Step3:  Defining Constants
Step4:  Model Agnostic Config for TFMA
Step5:  Fairness Callbacks and Computing Fairness Metrics
Step6:  Render Fairness Indicators
Render the Fairness Indicators widget with the exported evaluation results.
Below you will see bar charts displaying performance of each slice of the data on selected metrics. You can adjust the baseline comparison slice as well as the displayed threshold(s) using the drop down menus at the top of the visualization.
A relevant metric for this use case is true positive rate, also known as recall. Use the selector on the left hand side to choose the graph for true_positive_rate. These metric values match the values displayed on the model card.
For some photos, gender is labeled as young instead of male or female, if the person in the photo is too young to be accurately annotated.
<ASSISTANT_TASK:>
Python Code:
#@title Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# https://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
Explanation: Copyright 2020 The TensorFlow Authors.
End of explanation
!pip install apache_beam
!pip install fairness-indicators
!pip install witwidget
import os
import tempfile
import apache_beam as beam
import numpy as np
import pandas as pd
from datetime import datetime
import tensorflow_hub as hub
import tensorflow as tf
import tensorflow_model_analysis as tfma
import tensorflow_data_validation as tfdv
from tensorflow_model_analysis.addons.fairness.post_export_metrics import fairness_indicators
from tensorflow_model_analysis.addons.fairness.view import widget_view
from tensorflow_model_analysis.model_agnostic_eval import model_agnostic_predict as agnostic_predict
from tensorflow_model_analysis.model_agnostic_eval import model_agnostic_evaluate_graph
from tensorflow_model_analysis.model_agnostic_eval import model_agnostic_extractor
from witwidget.notebook.visualization import WitConfigBuilder
from witwidget.notebook.visualization import WitWidget
Explanation: FaceSSD Fairness Indicators Example Colab
<table class="tfo-notebook-buttons" align="left">
  <td>
    <a target="_blank" href="https://www.tensorflow.org/responsible_ai/fairness_indicators/tutorials/Facessd_Fairness_Indicators_Example_Colab"><img src="https://www.tensorflow.org/images/tf_logo_32px.png" />View on TensorFlow.org</a>
  </td>
  <td>
    <a target="_blank" href="https://colab.research.google.com/github/tensorflow/fairness-indicators/blob/master/g3doc/tutorials/Facessd_Fairness_Indicators_Example_Colab.ipynb"><img src="https://www.tensorflow.org/images/colab_logo_32px.png" />Run in Google Colab</a>
  </td>
  <td>
    <a target="_blank" href="https://github.com/tensorflow/fairness-indicators/tree/master/g3doc/tutorials/Facessd_Fairness_Indicators_Example_Colab.ipynb"><img src="https://www.tensorflow.org/images/GitHub-Mark-32px.png" />View on GitHub</a>
  </td>
  <td>
    <a href="https://storage.googleapis.com/tensorflow_docs/fairness-indicators/g3doc/tutorials/Facessd_Fairness_Indicators_Example_Colab.ipynb"><img src="https://www.tensorflow.org/images/download_logo_32px.png" />Download notebook</a>
  </td>
</table>
Overview
In this activity, you'll use Fairness Indicators to explore the FaceSSD predictions on Labeled Faces in the Wild dataset. Fairness Indicators is a suite of tools built on top of TensorFlow Model Analysis that enable regular evaluation of fairness metrics in product pipelines.
About the Dataset
In this exercise, you'll work with the FaceSSD prediction dataset, approximately 200k different image predictions and groundtruths generated by FaceSSD API.
About the Tools
TensorFlow Model Analysis is a library for evaluating both TensorFlow and non-TensorFlow machine learning models. It allows users to evaluate their models on large amounts of data in a distributed manner, computing in-graph and other metrics over different slices of data and visualize in notebooks.
TensorFlow Data Validation is one tool you can use to analyze your data. You can use it to find potential problems in your data, such as missing values and data imbalances, that can lead to Fairness disparities.
With Fairness Indicators, users will be able to: 
Evaluate model performance, sliced across defined groups of users
Feel confident about results with confidence intervals and evaluations at multiple thresholds
Importing
Run the following code to install the fairness_indicators library. This package contains the tools we'll be using in this exercise. Restart Runtime may be requested but is not necessary.
End of explanation
data_location = tf.keras.utils.get_file('lfw_dataset.tf', 'https://storage.googleapis.com/facessd_dataset/lfw_dataset.tfrecord')
stats = tfdv.generate_statistics_from_tfrecord(data_location=data_location)
tfdv.visualize_statistics(stats)
Explanation: Download and Understand the Data
Labeled Faces in the Wild is a public benchmark dataset for face verification, also known as pair matching. LFW contains more than 13,000 images of faces collected from the web.
We ran FaceSSD predictions on this dataset to predict whether a face is present in a given image. In this Colab, we will slice data according to gender to observe if there are any significant differences between model performance for different gender groups.
If there is more than one face in an image, gender is labeled as "MISSING".
We've hosted the dataset on Google Cloud Platform for convenience. Run the following code to download the data from GCP, the data will take about a minute to download and analyze.
End of explanation
BASE_DIR = tempfile.gettempdir()
tfma_eval_result_path = os.path.join(BASE_DIR, 'tfma_eval_result')
compute_confidence_intervals = True
slice_key = 'object/groundtruth/Gender'
label_key = 'object/groundtruth/face'
prediction_key = 'object/prediction/face'
feature_map = {
    slice_key:
        tf.io.FixedLenFeature([], tf.string, default_value=['none']),
    label_key:
        tf.io.FixedLenFeature([], tf.float32, default_value=[0.0]),
    prediction_key:
        tf.io.FixedLenFeature([], tf.float32, default_value=[0.0]),
}
Explanation: Defining Constants
End of explanation
model_agnostic_config = agnostic_predict.ModelAgnosticConfig(
    label_keys=[label_key],
    prediction_keys=[prediction_key],
    feature_spec=feature_map)
model_agnostic_extractors = [
    model_agnostic_extractor.ModelAgnosticExtractor(
        model_agnostic_config=model_agnostic_config, desired_batch_size=3),
    tfma.extractors.slice_key_extractor.SliceKeyExtractor(
          [tfma.slicer.SingleSliceSpec(),
           tfma.slicer.SingleSliceSpec(columns=[slice_key])])
]
Explanation: Model Agnostic Config for TFMA
End of explanation
# Helper class for counting examples in beam PCollection
class CountExamples(beam.CombineFn):
    def __init__(self, message):
      self.message = message
    def create_accumulator(self):
      return 0
    def add_input(self, current_sum, element):
      return current_sum + 1
    def merge_accumulators(self, accumulators): 
      return sum(accumulators)
    def extract_output(self, final_sum):
      if final_sum:
        print("%s: %d"%(self.message, final_sum))
metrics_callbacks = [
  tfma.post_export_metrics.fairness_indicators(
      thresholds=[0.1, 0.3, 0.5, 0.7, 0.9],
      labels_key=label_key,
      target_prediction_keys=[prediction_key]),
  tfma.post_export_metrics.auc(
      curve='PR',
      labels_key=label_key,
      target_prediction_keys=[prediction_key]),
]
eval_shared_model = tfma.types.EvalSharedModel(
    add_metrics_callbacks=metrics_callbacks,
    construct_fn=model_agnostic_evaluate_graph.make_construct_fn(
        add_metrics_callbacks=metrics_callbacks,
        config=model_agnostic_config))
with beam.Pipeline() as pipeline:
  # Read data.
  data = (
      pipeline
      | 'ReadData' >> beam.io.ReadFromTFRecord(data_location))
  # Count all examples.
  data_count = (
      data | 'Count number of examples' >> beam.CombineGlobally(
          CountExamples('Before filtering "Gender:MISSING"')))
  # If there are more than one face in image, the gender feature is 'MISSING'
  # and we are filtering that image out.
  def filter_missing_gender(element):
    example = tf.train.Example.FromString(element)
    if example.features.feature[slice_key].bytes_list.value[0] != b'MISSING':
      yield element
  filtered_data = (
      data
      | 'Filter Missing Gender' >> beam.ParDo(filter_missing_gender))
  # Count after filtering "Gender:MISSING".
  filtered_data_count = (
      filtered_data | 'Count number of examples after filtering'
      >> beam.CombineGlobally(
          CountExamples('After filtering "Gender:MISSING"')))
  # Because LFW data set has always faces by default, we are adding
  # labels as 1.0 for all images.
  def add_face_groundtruth(element):
    example = tf.train.Example.FromString(element)
    example.features.feature[label_key].float_list.value[:] = [1.0]
    yield example.SerializeToString()
  final_data = (
      filtered_data
      | 'Add Face Groundtruth' >> beam.ParDo(add_face_groundtruth))
  # Run TFMA.
  _ = (
      final_data
      | 'ExtractEvaluateAndWriteResults' >>
       tfma.ExtractEvaluateAndWriteResults(
                 eval_shared_model=eval_shared_model,
                 compute_confidence_intervals=compute_confidence_intervals,
                 output_path=tfma_eval_result_path,
                 extractors=model_agnostic_extractors))
eval_result = tfma.load_eval_result(output_path=tfma_eval_result_path)
Explanation: Fairness Callbacks and Computing Fairness Metrics
End of explanation
widget_view.render_fairness_indicator(eval_result=eval_result,
                                      slicing_column=slice_key)
Explanation: Render Fairness Indicators
Render the Fairness Indicators widget with the exported evaluation results.
Below you will see bar charts displaying performance of each slice of the data on selected metrics. You can adjust the baseline comparison slice as well as the displayed threshold(s) using the drop down menus at the top of the visualization.
A relevant metric for this use case is true positive rate, also known as recall. Use the selector on the left hand side to choose the graph for true_positive_rate. These metric values match the values displayed on the model card.
For some photos, gender is labeled as young instead of male or female, if the person in the photo is too young to be accurately annotated.
End of explanation
<END_TASK>