<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 Copyright 2021 The TensorFlow Authors.
Step1:  Human Pose Classification with MoveNet and TensorFlow Lite
This notebook teaches you how to train a pose classification model using MoveNet and TensorFlow Lite. It consists of 3 parts
Step9:  Code to run pose estimation using MoveNet
Step10:  Part 1
Step11:  Preprocess the TRAIN dataset
Step12:  Preprocess the TEST dataset
Step13:  Part 2
Step15:  Load the preprocessed CSVs into TRAIN and TEST datasets.
Step16:  Load and split the original TRAIN dataset into TRAIN (85% of the data) and VALIDATE (the remaining 15%).
Step21:  Define functions to convert the pose landmarks to a pose embedding (a.k.a. feature vector) for pose classification
Next, convert the landmark coordinates to a feature vector by
Step22:  Define a Keras model for pose classification
Our Keras model takes the detected pose landmarks, then calculates the pose embedding and predicts the pose class.
Step24:  Draw the confusion matrix to better understand the model performance
Step25:  (Optional) Investigate incorrect predictions
You can look at the poses from the TEST dataset that were incorrectly predicted to see whether the model accuracy can be improved.
Step26:  Part 3
Step27:  Then you'll write the label file which contains mapping from the class indexes to the human readable class names.
Step29:  As you've applied quantization to reduce the model size, let's evaluate the quantized TFLite model to check whether the accuracy drop is acceptable.
Step30:  Now you can download the TFLite model (model.tflite) and the label file (labels.txt) to classify custom poses. See the Android and Python/Raspberry Pi sample app for an end-to-end example of how to use the TFLite pose classification model.
<ASSISTANT_TASK:>
Python Code:
#@title Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# https://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
Explanation: Copyright 2021 The TensorFlow Authors.
End of explanation
!pip install -q opencv-python
import csv
import cv2
import itertools
import numpy as np
import pandas as pd
import os
import sys
import tempfile
import tqdm
from matplotlib import pyplot as plt
from matplotlib.collections import LineCollection
import tensorflow as tf
import tensorflow_hub as hub
from tensorflow import keras
from sklearn.model_selection import train_test_split
from sklearn.metrics import accuracy_score, classification_report, confusion_matrix
Explanation: Human Pose Classification with MoveNet and TensorFlow Lite
This notebook teaches you how to train a pose classification model using MoveNet and TensorFlow Lite. It consists of 3 parts:
* Part 1: Preprocess the pose classification training images into a CSV containing the landmarks detected from the images, and the ground truth labels.
* Part 2: Train a pose classification model that takes the landmark coordinates as input, and output the predicted labels.
* Part 3: Convert the pose classification model to TFLite.
<table class="tfo-notebook-buttons" align="left">
  <td>
    <a target="_blank" href="https://www.tensorflow.org/lite/tutorials/pose_classification"><img src="https://www.tensorflow.org/images/tf_logo_32px.png" />View on TensorFlow.org</a>
  </td>
  <td>
    <a target="_blank" href="https://colab.research.google.com/github/tensorflow/tensorflow/blob/master/tensorflow/lite/g3doc/tutorials/pose_classification.ipynb"><img src="https://www.tensorflow.org/images/colab_logo_32px.png" />Run in Google Colab</a>
  </td>
  <td>
    <a target="_blank" href="https://github.com/tensorflow/tensorflow/blob/master/tensorflow/lite/g3doc/tutorials/pose_classification.ipynb"><img src="https://www.tensorflow.org/images/GitHub-Mark-32px.png" />View source on GitHub</a>
  </td>
  <td>
    <a href="https://storage.googleapis.com/tensorflow_docs/tensorflow/tensorflow/lite/g3doc/tutorials/pose_classification.ipynb"><img src="https://www.tensorflow.org/images/download_logo_32px.png" />Download notebook</a>
  </td>
  <td>
    <a href="https://tfhub.dev/s?q=movenet"><img src="https://www.tensorflow.org/images/hub_logo_32px.png" />See TF Hub model</a>
  </td>
</table>
Preparation
In this step, you'll import the necessary libraries and define several functions to preprocess the training images into a CSV that contains the landmark coordinates and ground truth labels.
The input images needs to be put into a folder structure as below:
* Training images of each class should be stored in its own folder.
* The folder name should be the class name.
For example:
yoga_poses
|__ downdog
    |______ 00000128.jpg
    |______ 00000181.bmp
    |______ ...
|__ goddess
    |______ 00000243.jpg
    |______ 00000306.jpg
    |______ ...
...
The preprocessing logic outputs a CSV file that contains the pose landmarks detected by MoveNet from each image and the labels that can be used for training a pose classification model later.
If you only want to create the CSV file without knowing all the details, you can just run this section and proceed.
End of explanation
#@title Functions to run pose estimation with MoveNet
#@markdown You'll download the MoveNet Thunder model from [TensorFlow Hub](https://www.google.com/url?sa=D&q=https%3A%2F%2Ftfhub.dev%2Fs%3Fq%3Dmovenet), and reuse some inference and visualization logic from the [MoveNet Raspberry Pi (Python)](https://github.com/tensorflow/examples/tree/master/lite/examples/pose_estimation/raspberry_pi) sample app to detect landmarks (ear, nose, wrist etc.) from the input images.
#@markdown *Note: You should use the most accurate pose estimation model (i.e. MoveNet Thunder) to detect the keypoints and use them to train the pose classification model to achieve the best accuracy. When running inference, you can use a pose estimation model of your choice (e.g. either MoveNet Lightning or Thunder).*
# Download model from TF Hub and check out inference code from GitHub
!wget -q -O movenet_thunder.tflite https://tfhub.dev/google/lite-model/movenet/singlepose/thunder/tflite/float16/4?lite-format=tflite
!git clone https://github.com/tensorflow/examples.git
pose_sample_rpi_path = os.path.join(os.getcwd(), 'examples/lite/examples/pose_estimation/raspberry_pi')
sys.path.append(pose_sample_rpi_path)
# Load MoveNet Thunder model
import utils
from movenet import Movenet
movenet = Movenet('movenet_thunder')
# Define function to run pose estimation using MoveNet Thunder.
# You'll apply MoveNet's cropping algorithm and run inference multiple times on
# the input image to improve pose estimation accuracy.
def detect(input_tensor, inference_count=3):
  Runs detection on an input image.
 
  Args:
    input_tensor: A [1, height, width, 3] Tensor of type tf.float32.
      Note that height and width can be anything since the image will be
      immediately resized according to the needs of the model within this
      function.
    inference_count: Number of times the model should run repeatly on the
      same input image to improve detection accuracy.
 
  Returns:
    A dict containing 1 Tensor of shape [1, 1, 17, 3] representing the
    keypoint coordinates and scores.
  
  image_height, image_width, channel = input_tensor.shape
 
  # Detect pose using the full input image
  movenet.detect(input_tensor.numpy(), reset_crop_region=True)
 
  # Repeatedly using previous detection result to identify the region of
  # interest and only croping that region to improve detection accuracy
  for _ in range(inference_count - 1):
    keypoint_with_scores = movenet.detect(input_tensor.numpy(), 
                                          reset_crop_region=False)
  return keypoint_with_scores
#@title Functions to visualize the pose estimation results.
def draw_prediction_on_image(
    image, keypoints_with_scores, crop_region=None, close_figure=True,
    keep_input_size=False):
  Draws the keypoint predictions on image.
 
  Args:
    image: An numpy array with shape [height, width, channel] representing the
      pixel values of the input image.
    keypoints_with_scores: An numpy array with shape [1, 1, 17, 3] representing
      the keypoint coordinates and scores returned from the MoveNet model.
    crop_region: Set the region to crop the output image.
    close_figure: Whether to close the plt figure after the function returns.
    keep_input_size: Whether to keep the size of the input image.
 
  Returns:
    An numpy array with shape [out_height, out_width, channel] representing the
    image overlaid with keypoint predictions.
  
  height, width, channel = image.shape
  aspect_ratio = float(width) / height
  fig, ax = plt.subplots(figsize=(12 * aspect_ratio, 12))
  # To remove the huge white borders
  fig.tight_layout(pad=0)
  ax.margins(0)
  ax.set_yticklabels([])
  ax.set_xticklabels([])
  plt.axis('off')
 
  im = ax.imshow(image)
  line_segments = LineCollection([], linewidths=(2), linestyle='solid')
  ax.add_collection(line_segments)
  # Turn off tick labels
  scat = ax.scatter([], [], s=60, color='#FF1493', zorder=2)
 
  # Calculate visualization items from pose estimation result
  (keypoint_locs, keypoint_edges,
   edge_colors) = utils.keypoints_and_edges_for_display(
       keypoints_with_scores, height, width) 
  edge_colors = [(r/255.0, g/255.0, b/255.0) for (r ,g , b) in edge_colors]
   
  line_segments.set_segments(keypoint_edges)
  line_segments.set_color(edge_colors)
  if keypoint_edges.shape[0]:
    line_segments.set_segments(keypoint_edges)
    line_segments.set_color(edge_colors)
  if keypoint_locs.shape[0]:
    scat.set_offsets(keypoint_locs)
 
  if crop_region is not None:
    xmin = max(crop_region['x_min'] * width, 0.0)
    ymin = max(crop_region['y_min'] * height, 0.0)
    rec_width = min(crop_region['x_max'], 0.99) * width - xmin
    rec_height = min(crop_region['y_max'], 0.99) * height - ymin
    rect = patches.Rectangle(
        (xmin,ymin),rec_width,rec_height,
        linewidth=1,edgecolor='b',facecolor='none')
    ax.add_patch(rect)
 
  fig.canvas.draw()
  image_from_plot = np.frombuffer(fig.canvas.tostring_rgb(), dtype=np.uint8)
  image_from_plot = image_from_plot.reshape(
      fig.canvas.get_width_height()[::-1] + (3,))
 
  if close_figure:
    plt.close(fig)
 
  if keep_input_size:
    image_from_plot = cv2.resize(image_from_plot, dsize=(width, height),
                                 interpolation=cv2.INTER_CUBIC)
      
  return image_from_plot
#@title Code to load the images, detect pose landmarks and save them into a CSV file
class MoveNetPreprocessor(object):
  Helper class to preprocess pose sample images for classification.
 
  def __init__(self,
               images_in_folder,
               images_out_folder,
               csvs_out_path):
    Creates a preprocessor to detection pose from images and save as CSV.
    Args:
      images_in_folder: Path to the folder with the input images. It should
        follow this structure:
        yoga_poses
        |__ downdog
            |______ 00000128.jpg
            |______ 00000181.bmp
            |______ ...
        |__ goddess
            |______ 00000243.jpg
            |______ 00000306.jpg
            |______ ...
        ...
      images_out_folder: Path to write the images overlay with detected
        landmarks. These images are useful when you need to debug accuracy
        issues.
      csvs_out_path: Path to write the CSV containing the detected landmark
        coordinates and label of each image that can be used to train a pose
        classification model.
    
    self._images_in_folder = images_in_folder
    self._images_out_folder = images_out_folder
    self._csvs_out_path = csvs_out_path
    self._messages = []
    # Create a temp dir to store the pose CSVs per class
    self._csvs_out_folder_per_class = tempfile.mkdtemp()
 
    # Get list of pose classes and print image statistics
    self._pose_class_names = sorted(
        [n for n in os.listdir(self._images_in_folder) if not n.startswith('.')]
        )
    
  def process(self, per_pose_class_limit=None, detection_threshold=0.1):
    Preprocesses images in the given folder.
    Args:
      per_pose_class_limit: Number of images to load. As preprocessing usually
        takes time, this parameter can be specified to make the reduce the
        dataset for testing.
      detection_threshold: Only keep images with all landmark confidence score
        above this threshold.
    
    # Loop through the classes and preprocess its images
    for pose_class_name in self._pose_class_names:
      print('Preprocessing', pose_class_name, file=sys.stderr)
      # Paths for the pose class.
      images_in_folder = os.path.join(self._images_in_folder, pose_class_name)
      images_out_folder = os.path.join(self._images_out_folder, pose_class_name)
      csv_out_path = os.path.join(self._csvs_out_folder_per_class,
                                  pose_class_name + '.csv')
      if not os.path.exists(images_out_folder):
        os.makedirs(images_out_folder)
 
      # Detect landmarks in each image and write it to a CSV file
      with open(csv_out_path, 'w') as csv_out_file:
        csv_out_writer = csv.writer(csv_out_file, 
                                    delimiter=',', 
                                    quoting=csv.QUOTE_MINIMAL)
        # Get list of images
        image_names = sorted(
            [n for n in os.listdir(images_in_folder) if not n.startswith('.')])
        if per_pose_class_limit is not None:
          image_names = image_names[:per_pose_class_limit]
 
        # Detect pose landmarks from each image
        for image_name in tqdm.tqdm(image_names):
          image_path = os.path.join(images_in_folder, image_name)
          try:
            image = tf.io.read_file(image_path)
            image = tf.io.decode_jpeg(image)
          except:
            self._messages.append('Skipped ' + image_path + '. Invalid image.')
            continue
          else:
            image = tf.io.read_file(image_path)
            image = tf.io.decode_jpeg(image)
            image_height, image_width, channel = image.shape
          
          # Skip images that isn't RGB because Movenet requires RGB images
          if channel != 3:
            self._messages.append('Skipped ' + image_path +
                                  '. Image isn\'t in RGB format.')
            continue
          keypoint_with_scores = detect(image)
          
          # Save landmarks if all landmarks were detected
          min_landmark_score = np.amin(keypoint_with_scores[:2])
          should_keep_image = min_landmark_score >= detection_threshold
          if not should_keep_image:
            self._messages.append('Skipped ' + image_path +
                                  '. No pose was confidentlly detected.')
            continue
          # Draw the prediction result on top of the image for debugging later
          output_overlay = draw_prediction_on_image(
              image.numpy().astype(np.uint8), keypoint_with_scores, 
              crop_region=None, close_figure=True, keep_input_size=True)
        
          # Write detection result to into an image file
          output_frame = cv2.cvtColor(output_overlay, cv2.COLOR_RGB2BGR)
          cv2.imwrite(os.path.join(images_out_folder, image_name), output_frame)
        
          # Get landmarks and scale it to the same size as the input image
          pose_landmarks = np.array(
              [[lmk[0] * image_width, lmk[1] * image_height, lmk[2]]
                for lmk in keypoint_with_scores],
              dtype=np.float32)
          # Write the landmark coordinates to its per-class CSV file
          coordinates = pose_landmarks.flatten().astype(np.str).tolist()
          csv_out_writer.writerow([image_name] + coordinates)
      
    # Print the error message collected during preprocessing.
    print('\n'.join(self._messages))
    # Combine all per-class CSVs into a single output file
    all_landmarks_df = self._all_landmarks_as_dataframe()
    all_landmarks_df.to_csv(self._csvs_out_path, index=False)
  def class_names(self):
    List of classes found in the training dataset.
    return self._pose_class_names
  
  def _all_landmarks_as_dataframe(self):
    Merge all per-class CSVs into a single dataframe.
    total_df = None
    for class_index, class_name in enumerate(self._pose_class_names):
      csv_out_path = os.path.join(self._csvs_out_folder_per_class,
                                  class_name + '.csv')
      
      per_class_df = pd.read_csv(csv_out_path, header=None)
      
      # Add the labels
      per_class_df['class_no'] = [class_index]*len(per_class_df)
      per_class_df['class_name'] = [class_name]*len(per_class_df)
      # Append the folder name to the filename column (first column)
      per_class_df[per_class_df.columns[0]] = (os.path.join(class_name, '') 
        + per_class_df[per_class_df.columns[0]].astype(str))
      if total_df is None:
        # For the first class, assign its data to the total dataframe
        total_df = per_class_df
      else:
        # Concatenate each class's data into the total dataframe
        total_df = pd.concat([total_df, per_class_df], axis=0)
 
    list_name = [[key + '_x', key + '_y', key + '_score'] 
                 for key in utils.KEYPOINT_DICT.keys()] 
    header_name = []
    for columns_name in list_name:
      header_name += columns_name
    header_name = ['file_name'] + header_name
    header_map = {total_df.columns[i]: header_name[i] 
                  for i in range(len(header_name))}
 
    total_df.rename(header_map, axis=1, inplace=True)
    return total_df
#@title (Optional) Code snippet to try out the Movenet pose estimation logic
#@markdown You can download an image from the internet, run the pose estimation logic on it and plot the detected landmarks on top of the input image. 
#@markdown *Note: This code snippet is also useful for debugging when you encounter an image with bad pose classification accuracy. You can run pose estimation on the image and see if the detected landmarks look correct or not before investigating the pose classification logic.*
test_image_url = "https://cdn.pixabay.com/photo/2017/03/03/17/30/yoga-2114512_960_720.jpg" #@param {type:"string"}
!wget -O /tmp/image.jpeg {test_image_url}
if len(test_image_url):
  image = tf.io.read_file('/tmp/image.jpeg')
  image = tf.io.decode_jpeg(image)
  keypoint_with_scores = detect(image)
  _ = draw_prediction_on_image(image, keypoint_with_scores, crop_region=None, 
                               close_figure=False, keep_input_size=True)
Explanation: Code to run pose estimation using MoveNet
End of explanation
!wget -O yoga_poses.zip http://download.tensorflow.org/data/pose_classification/yoga_poses.zip
!unzip -q yoga_poses.zip -d yoga_cg
Explanation: Part 1: Preprocess the input images
You'll use the functions defined earlier to preprocess the input images into a CSV file that contains the detected landmarks and ground truth labels. You'll also plot the pose landmarks onto the input image to make debugging easier later on.
The code snippet used to preprocess the input image is as below.
```python
images_in_folder = 'yoga_poses'
images_out_folder = 'yoga_poses_with_landmarks'
csvs_out_path = 'landmarks.csv'
Create a preprocessor object
preprocessor = MoveNetPreprocessor(
    images_in_folder=images_in_folder,
    images_out_folder=images_out_folder,
    csvs_out_path=csvs_out_path,
)
Start preprocessing the input images.
You can set per_pose_class_limit to a small number for debugging.
preprocessor.process(per_pose_class_limit=None)
```
In this tutorial, you'll use a CG-generated yoga pose dataset. It contains images of multiple CG-generated models doing 5 different yoga poses. There's a TRAIN dataset and a TEST dataset. You start by downloading the dataset to Colab and preprocess it.
Note: It takes about 15 minutes to finish this preprocessing step. If you don't want to wait and just want to run through this tutorial, you can skip to step 2, uncomment the code in (Optional) Download the preprocessed dataset if you didn't run part 1 section to download the CSV files which are the same as those will be created in this preprocessing step.
End of explanation
images_in_train_folder = 'yoga_cg/train'
images_out_train_folder = 'poses_images_out_train'
csvs_out_train_path = 'train_data.csv'
preprocessor = MoveNetPreprocessor(
    images_in_folder=images_in_train_folder,
    images_out_folder=images_out_train_folder,
    csvs_out_path=csvs_out_train_path,
)
preprocessor.process(per_pose_class_limit=None)
Explanation: Preprocess the TRAIN dataset
End of explanation
images_in_test_folder = 'yoga_cg/test'
images_out_test_folder = 'poses_images_out_test'
csvs_out_test_path = 'test_data.csv'
preprocessor = MoveNetPreprocessor(
    images_in_folder=images_in_test_folder,
    images_out_folder=images_out_test_folder,
    csvs_out_path=csvs_out_test_path,
)
preprocessor.process(per_pose_class_limit=None)
Explanation: Preprocess the TEST dataset
End of explanation
# # Download the preprocessed CSV so that you don't need to run the part 1 again.
# !wget -O train_data.csv http://download.tensorflow.org/data/pose_classification/yoga_train_data.csv
# !wget -O test_data.csv http://download.tensorflow.org/data/pose_classification/yoga_test_data.csv
# csvs_out_train_path = 'train_data.csv'
# csvs_out_test_path = 'test_data.csv'
Explanation: Part 2: Train a pose classification model that takes the landmark coordinates as input, and output the predicted labels.
You'll build a TensorFlow model that takes the landmark coordinates and predicts the pose class that the person in the input image performs. The model consists of two submodels:
Submodel 1 calculates a pose embedding (a.k.a feature vector) from the detected landmark coordinates.
Submodel 2 feeds pose embedding through several Dense layer to predict the pose class.
You'll then train the model based on the dataset that were preprocessed in part 1.
(Optional) Download the preprocessed dataset if you didn't run part 1
End of explanation
def load_pose_landmarks(csv_path):
  Loads a CSV created by from the MoveNetPreprocessor.
  
  Returns:
    X: Detected landmark coordinates and scores of shape (N, 17 * 3)
    y: Ground truth labels of shape of shape (N, label_count)
    classes: The list of all class names found in the dataset
    dataframe: The CSV loaded as a Pandas dataframe features (X) and ground
      truth labels (y) to use later to train a pose classification model.
  
  # Load the CSV file
  dataframe = pd.read_csv(csv_path)
  df_to_process = dataframe.copy()
  # Drop the file_name columns as you don't need it during training.
  df_to_process.drop(columns=['file_name'], inplace=True)
  # Extract the list of class names
  classes = df_to_process.pop('class_name').unique()
  # Extract the labels
  y = df_to_process.pop('class_no')
  # Convert the input features and labels into the correct format for training.
  X = df_to_process.astype('float64')
  y = keras.utils.to_categorical(y)
  return X, y, classes, dataframe
Explanation: Load the preprocessed CSVs into TRAIN and TEST datasets.
End of explanation
# Load the train data
X, y, class_names, _ = load_pose_landmarks(csvs_out_train_path)
# Split training data (X, y) into (X_train, y_train) and (X_val, y_val)
X_train, X_val, y_train, y_val = train_test_split(X, y,
                                                  test_size=0.15)
# Load the test data
X_test, y_test, _, df_test = load_pose_landmarks(csvs_out_test_path)
Explanation: Load and split the original TRAIN dataset into TRAIN (85% of the data) and VALIDATE (the remaining 15%).
End of explanation
def get_center_point(landmarks, left_name, right_name):
  Calculates the center point of the two given landmarks.
  left = tf.gather(landmarks, utils.KEYPOINT_DICT[left_name], axis=1)
  right = tf.gather(landmarks, utils.KEYPOINT_DICT[right_name], axis=1)
  center = left * 0.5 + right * 0.5
  return center
def get_pose_size(landmarks, torso_size_multiplier=2.5):
  Calculates pose size.
  It is the maximum of two values:
    * Torso size multiplied by `torso_size_multiplier`
    * Maximum distance from pose center to any pose landmark
  
  # Hips center
  hips_center = get_center_point(landmarks, "left_hip", "right_hip")
  # Shoulders center
  shoulders_center = get_center_point(landmarks, 
                                      "left_shoulder", "right_shoulder")
  # Torso size as the minimum body size
  torso_size = tf.linalg.norm(shoulders_center - hips_center)
  # Pose center
  pose_center_new = get_center_point(landmarks, "left_hip", "right_hip")
  pose_center_new = tf.expand_dims(pose_center_new, axis=1)
  # Broadcast the pose center to the same size as the landmark vector to
  # perform substraction
  pose_center_new = tf.broadcast_to(pose_center_new,
                                    [tf.size(landmarks) // (17*2), 17, 2])
  # Dist to pose center
  d = tf.gather(landmarks - pose_center_new, 0, axis=0,
                name="dist_to_pose_center")
  # Max dist to pose center
  max_dist = tf.reduce_max(tf.linalg.norm(d, axis=0))
  # Normalize scale
  pose_size = tf.maximum(torso_size * torso_size_multiplier, max_dist)
  return pose_size
def normalize_pose_landmarks(landmarks):
  Normalizes the landmarks translation by moving the pose center to (0,0) and
  scaling it to a constant pose size.
  
  # Move landmarks so that the pose center becomes (0,0)
  pose_center = get_center_point(landmarks, "left_hip", "right_hip")
  pose_center = tf.expand_dims(pose_center, axis=1)
  # Broadcast the pose center to the same size as the landmark vector to perform
  # substraction
  pose_center = tf.broadcast_to(pose_center, 
                                [tf.size(landmarks) // (17*2), 17, 2])
  landmarks = landmarks - pose_center
  # Scale the landmarks to a constant pose size
  pose_size = get_pose_size(landmarks)
  landmarks /= pose_size
  return landmarks
def landmarks_to_embedding(landmarks_and_scores):
  Converts the input landmarks into a pose embedding.
  # Reshape the flat input into a matrix with shape=(17, 3)
  reshaped_inputs = keras.layers.Reshape((17, 3))(landmarks_and_scores)
  # Normalize landmarks 2D
  landmarks = normalize_pose_landmarks(reshaped_inputs[:, :, :2])
  # Flatten the normalized landmark coordinates into a vector
  embedding = keras.layers.Flatten()(landmarks)
  return embedding
Explanation: Define functions to convert the pose landmarks to a pose embedding (a.k.a. feature vector) for pose classification
Next, convert the landmark coordinates to a feature vector by:
1. Moving the pose center to the origin.
2. Scaling the pose so that the pose size becomes 1
3. Flattening these coordinates into a feature vector
Then use this feature vector to train a neural-network based pose classifier.
End of explanation
# Define the model
inputs = tf.keras.Input(shape=(51))
embedding = landmarks_to_embedding(inputs)
layer = keras.layers.Dense(128, activation=tf.nn.relu6)(embedding)
layer = keras.layers.Dropout(0.5)(layer)
layer = keras.layers.Dense(64, activation=tf.nn.relu6)(layer)
layer = keras.layers.Dropout(0.5)(layer)
outputs = keras.layers.Dense(5, activation="softmax")(layer)
model = keras.Model(inputs, outputs)
model.summary()
model.compile(
    optimizer='adam',
    loss='categorical_crossentropy',
    metrics=['accuracy']
)
# Add a checkpoint callback to store the checkpoint that has the highest
# validation accuracy.
checkpoint_path = "weights.best.hdf5"
checkpoint = keras.callbacks.ModelCheckpoint(checkpoint_path,
                             monitor='val_accuracy',
                             verbose=1,
                             save_best_only=True,
                             mode='max')
earlystopping = keras.callbacks.EarlyStopping(monitor='val_accuracy', 
                                              patience=20)
# Start training
history = model.fit(X_train, y_train,
                    epochs=200,
                    batch_size=16,
                    validation_data=(X_val, y_val),
                    callbacks=[checkpoint, earlystopping])
# Visualize the training history to see whether you're overfitting.
plt.plot(history.history['accuracy'])
plt.plot(history.history['val_accuracy'])
plt.title('Model accuracy')
plt.ylabel('accuracy')
plt.xlabel('epoch')
plt.legend(['TRAIN', 'VAL'], loc='lower right')
plt.show()
# Evaluate the model using the TEST dataset
loss, accuracy = model.evaluate(X_test, y_test)
Explanation: Define a Keras model for pose classification
Our Keras model takes the detected pose landmarks, then calculates the pose embedding and predicts the pose class.
End of explanation
def plot_confusion_matrix(cm, classes,
                          normalize=False,
                          title='Confusion matrix',
                          cmap=plt.cm.Blues):
  Plots the confusion matrix.
  if normalize:
    cm = cm.astype('float') / cm.sum(axis=1)[:, np.newaxis]
    print("Normalized confusion matrix")
  else:
    print('Confusion matrix, without normalization')
  plt.imshow(cm, interpolation='nearest', cmap=cmap)
  plt.title(title)
  plt.colorbar()
  tick_marks = np.arange(len(classes))
  plt.xticks(tick_marks, classes, rotation=55)
  plt.yticks(tick_marks, classes)
  fmt = '.2f' if normalize else 'd'
  thresh = cm.max() / 2.
  for i, j in itertools.product(range(cm.shape[0]), range(cm.shape[1])):
    plt.text(j, i, format(cm[i, j], fmt),
              horizontalalignment="center",
              color="white" if cm[i, j] > thresh else "black")
  plt.ylabel('True label')
  plt.xlabel('Predicted label')
  plt.tight_layout()
# Classify pose in the TEST dataset using the trained model
y_pred = model.predict(X_test)
# Convert the prediction result to class name
y_pred_label = [class_names[i] for i in np.argmax(y_pred, axis=1)]
y_true_label = [class_names[i] for i in np.argmax(y_test, axis=1)]
# Plot the confusion matrix
cm = confusion_matrix(np.argmax(y_test, axis=1), np.argmax(y_pred, axis=1))
plot_confusion_matrix(cm,
                      class_names,
                      title ='Confusion Matrix of Pose Classification Model')
# Print the classification report
print('\nClassification Report:\n', classification_report(y_true_label,
                                                          y_pred_label))
Explanation: Draw the confusion matrix to better understand the model performance
End of explanation
IMAGE_PER_ROW = 3
MAX_NO_OF_IMAGE_TO_PLOT = 30
# Extract the list of incorrectly predicted poses
false_predict = [id_in_df for id_in_df in range(len(y_test)) \
                 if y_pred_label[id_in_df] != y_true_label[id_in_df]]
if len(false_predict) > MAX_NO_OF_IMAGE_TO_PLOT:
  false_predict = false_predict[:MAX_NO_OF_IMAGE_TO_PLOT]
# Plot the incorrectly predicted images
row_count = len(false_predict) // IMAGE_PER_ROW + 1
fig = plt.figure(figsize=(10 * IMAGE_PER_ROW, 10 * row_count))
for i, id_in_df in enumerate(false_predict):
  ax = fig.add_subplot(row_count, IMAGE_PER_ROW, i + 1)
  image_path = os.path.join(images_out_test_folder,
                            df_test.iloc[id_in_df]['file_name'])
  image = cv2.imread(image_path)
  plt.title("Predict: %s; Actual: %s"
            % (y_pred_label[id_in_df], y_true_label[id_in_df]))
  plt.imshow(cv2.cvtColor(image, cv2.COLOR_BGR2RGB))
plt.show()
Explanation: (Optional) Investigate incorrect predictions
You can look at the poses from the TEST dataset that were incorrectly predicted to see whether the model accuracy can be improved.
End of explanation
converter = tf.lite.TFLiteConverter.from_keras_model(model)
converter.optimizations = [tf.lite.Optimize.DEFAULT]
tflite_model = converter.convert()
print('Model size: %dKB' % (len(tflite_model) / 1024))
with open('model.tflite', 'wb') as f:
  f.write(tflite_model)
Explanation: Part 3: Convert the pose classification model to TensorFlow Lite
You'll convert the Keras pose classification model to the TensorFlow Lite format so that you can deploy it to mobile apps, web browsers and IoT devices. When converting the model, you'll apply dynamic range quantization to reduce the pose classification TensorFlow Lite model size by about 4 times with insignificant accuracy loss.
Note: TensorFlow Lite supports multiple quantization schemes. See the documentation if you are interested to learn more.
End of explanation
with open('labels.txt', 'w') as f:
  f.write('\n'.join(class_names))
Explanation: Then you'll write the label file which contains mapping from the class indexes to the human readable class names.
End of explanation
def evaluate_model(interpreter, X, y_true):
  Evaluates the given TFLite model and return its accuracy.
  input_index = interpreter.get_input_details()[0]["index"]
  output_index = interpreter.get_output_details()[0]["index"]
  # Run predictions on all given poses.
  y_pred = []
  for i in range(len(y_true)):
    # Pre-processing: add batch dimension and convert to float32 to match with
    # the model's input data format.
    test_image = X[i: i + 1].astype('float32')
    interpreter.set_tensor(input_index, test_image)
    # Run inference.
    interpreter.invoke()
    # Post-processing: remove batch dimension and find the class with highest
    # probability.
    output = interpreter.tensor(output_index)
    predicted_label = np.argmax(output()[0])
    y_pred.append(predicted_label)
  # Compare prediction results with ground truth labels to calculate accuracy.
  y_pred = keras.utils.to_categorical(y_pred)
  return accuracy_score(y_true, y_pred)
# Evaluate the accuracy of the converted TFLite model
classifier_interpreter = tf.lite.Interpreter(model_content=tflite_model)
classifier_interpreter.allocate_tensors()
print('Accuracy of TFLite model: %s' %
      evaluate_model(classifier_interpreter, X_test, y_test))
Explanation: As you've applied quantization to reduce the model size, let's evaluate the quantized TFLite model to check whether the accuracy drop is acceptable.
End of explanation
!zip classifier.zip labels.txt model.tflite
# Download the zip archive if running on Colab.
try:
  from google.colab import files
  files.download('classifier.zip') 
except:
  pass
Explanation: Now you can download the TFLite model (model.tflite) and the label file (labels.txt) to classify custom poses. See the Android and Python/Raspberry Pi sample app for an end-to-end example of how to use the TFLite pose classification model.
End of explanation
<END_TASK>