<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 Copyright 2021 Google LLC
Licensed under the Apache License, Version 2.0 (the "License")
Step1:  Retrain EfficientDet for the Edge TPU with TensorFlow Lite Model Maker
In this tutorial, we'll retrain the EfficientDet-Lite object detection model (derived from EfficientDet) using the TensorFlow Lite Model Maker library, and then compile it to run on the Coral Edge TPU. All in about 30 minutes.
By default, w'll retrain the model using a publicly available dataset of salad photos, teaching the model to recognize a salad and some of the ingredients. But we've also provided code so you can upload your own training dataset in the Pascal VOC XML format.
Here's an example of the salad training results
Step2:  Load the training data
To use the default salad training dataset, just run all the code below as-is.
But if you want to train with your own image dataset, follow these steps
Step3:  Load the salads CSV dataset
Model Maker requires that we load our dataset using the DataLoader API. So in this case, we'll load it from a CSV file that defines 175 images for training, 25 images for validation, and 25 images for testing.
Step4:  If you want to load your own dataset as a CSV file, you can learn more about the format in Formatting a training data CSV. You can load your CSV either from Cloud Storage (as shown above) or from a local path.
DataLoader can also load your dataset in other formats, such as from a set of TFRecord files or from a local directory using the Pascal VOC format (shown below for a custom dataset). 
(Optional) Load your own Pascal VOC dataset
To use your custom dataset, you need to modify a few variables here, such as your ZIP filename, your label map, and the path to your images/annotations
Step6:  Now you're ready to train the model with your custom dataset. But before you run the notebook, you should also skip to the Export to TensorFlow Lite section and change the TFLITE_FILENAME and LABLES_FILENAME for your exported files.
Then run the whole notebook by clicking Runtime > Run all.
Step7:  Select the model spec
Model Maker supports the EfficientDet-Lite family of object detection models that are compatible with the Edge TPU. (EfficientDet-Lite is derived from EfficientDet, which offers state-of-the-art accuracy in a small model size). There are several model sizes you can choose from
Step8:  The EfficientDetLite0Spec constructor also supports several arguments that specify training options, such as the max number of detections (default is 25 for the TF Lite model) and whether to use Cloud TPUs for training. You can also use the constructor to specify the number of training epochs and the batch size, but you can also specify those in the next step.
Create and train the model
Now we need to create our model according to the model spec, load our dataset into the model, specify training parameters, and begin training. 
Using Model Maker, we accomplished all of that with create()
Step9:  Evaluate the model
Now we'll use the test dataset to evaluate how well the model performs with data it has never seen before.
The evaluate() method provides output in the style of COCO evaluation metrics
Step10:  Because the default batch size for EfficientDetLite models is 64, this needs only 1 step to go through all 25 images in the salad test set. You can also specify the batch_size argument when you call evaluate().
Export to TensorFlow Lite
Next, we'll export the model to the TensorFlow Lite format. By default, the export() method performs full integer post-training quantization, which is exactly what we need for compatibility with the Edge TPU. (Model Maker uses the same dataset we gave to our model spec as a representative dataset, which is required for full-int quantization.)
We just need to specify the export directory and format. By default, it exports to TF Lite, but we also want a labels file, so we declare both
Step11:  Evaluate the TF Lite model
Exporting the model to TensorFlow Lite can affect the model accuracy, due to the reduced numerical precision from quantization and because the original TensorFlow model uses per-class non-max supression (NMS) for post-processing, while the TF Lite model uses global NMS, which is faster but less accurate.
Therefore you should always evaluate the exported TF Lite model and be sure it still meets your requirements
Step12:  Try the TFLite model
Just to be sure of things, let's run the model ourselves with an image from the test set.
Step14:  To simplify our code, we'll use the PyCoral API
Step15:  Compile for the Edge TPU
First we need to download the Edge TPU Compiler
Step16:  Before compiling the .tflite file for the Edge TPU, it's important to consider whether your model will fit into the Edge TPU memory. 
The Edge TPU has approximately 8 MB of SRAM for caching model paramaters, so any model close to or over 8 MB will not fit onto the Edge TPU memory. That means the inference times are longer, because some model parameters must be fetched from the host system memory.
One way to elimiate the extra latency is to use model pipelining, which splits the model into segments that can run on separate Edge TPUs in series. This can significantly reduce the latency for big models.
The following table provides recommendations for the number of Edge TPUs to use with each EfficientDet-Lite model.
| Model architecture | Minimum TPUs | Recommended TPUs
|--------------------|-------|-------|
| EfficientDet-Lite0 | 1     | 1     |
| EfficientDet-Lite1 | 1     | 1     |
| EfficientDet-Lite2 | 1     | 2     |
| EfficientDet-Lite3 | 2     | 2     |
| EfficientDet-Lite4 | 2     | 3     |
If you need extra Edge TPUs for your model, then update NUMBER_OF_TPUS here
Step17:  Beware when using multiple segments
<ASSISTANT_TASK:>
Python Code:
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# https://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
Explanation: Copyright 2021 Google LLC
Licensed under the Apache License, Version 2.0 (the "License")
End of explanation
!pip install -q tflite-model-maker
import numpy as np
import os
from tflite_model_maker.config import ExportFormat
from tflite_model_maker import model_spec
from tflite_model_maker import object_detector
import tensorflow as tf
assert tf.__version__.startswith('2')
tf.get_logger().setLevel('ERROR')
from absl import logging
logging.set_verbosity(logging.ERROR)
Explanation: Retrain EfficientDet for the Edge TPU with TensorFlow Lite Model Maker
In this tutorial, we'll retrain the EfficientDet-Lite object detection model (derived from EfficientDet) using the TensorFlow Lite Model Maker library, and then compile it to run on the Coral Edge TPU. All in about 30 minutes.
By default, w'll retrain the model using a publicly available dataset of salad photos, teaching the model to recognize a salad and some of the ingredients. But we've also provided code so you can upload your own training dataset in the Pascal VOC XML format.
Here's an example of the salad training results:
<img src="https://storage.googleapis.com/site_and_emails_static_assets/Images/efficientdet-salads.png?" width="400" hspace="0">
<a href="https://colab.research.google.com/github/google-coral/tutorials/blob/master/retrain_efficientdet_model_maker_tf2.ipynb" target="_parent"><img src="https://colab.research.google.com/assets/colab-badge.svg" alt="Open in Colab"></a>
&nbsp;&nbsp;&nbsp;&nbsp;
<a href="https://github.com/google-coral/tutorials/blob/master/retrain_efficientdet_model_maker_tf2.ipynb" target="_parent"><img src="https://img.shields.io/static/v1?logo=GitHub&label=&color=333333&style=flat&message=View%20on%20GitHub" alt="View in GitHub"></a>
If you want to run the notebook with the salad dataset, you can run the whole thing now by clicking Runtime > Run all in the Colab toolbar. But if you want to use your own dataset, then continue down to Load the training data and follow the instructions there.
Note: If using a custom dataset, beware that if your dataset includes more than 20 classes, you'll probably have slower inference speeds compared to if you have fewer classes. This is due to an aspect of the EfficientDet architecture in which a certain layer cannot compile for the Edge TPU when it carries more than 20 classes.
Import the required packages
End of explanation
use_custom_dataset = False #@param ["False", "True"] {type:"raw"}
dataset_is_split = False #@param ["False", "True"] {type:"raw"}
Explanation: Load the training data
To use the default salad training dataset, just run all the code below as-is.
But if you want to train with your own image dataset, follow these steps:
Be sure your dataset is annotated in Pascal VOC XML (various tools can help create VOC annotations, such as LabelImg). Then create a ZIP file with all your JPG images and XML files (JPG and XML files can all be in one directory or in separate directories).
Click the Files tab in the left panel and just drag-drop your ZIP file there to upload it.
Use the following drop-down option to set use_custom_dataset to True.
If your dataset is already split into separate directories for training, validation, and testing, also set dataset_is_split to True. (If your dataset is not split, leave it False and we'll split it below.)
Then skip to Load your own Pascal VOC dataset and follow the rest of the instructions there.
End of explanation
if not use_custom_dataset:
  train_data, validation_data, test_data = object_detector.DataLoader.from_csv('gs://cloud-ml-data/img/openimage/csv/salads_ml_use.csv')
Explanation: Load the salads CSV dataset
Model Maker requires that we load our dataset using the DataLoader API. So in this case, we'll load it from a CSV file that defines 175 images for training, 25 images for validation, and 25 images for testing.
End of explanation
if use_custom_dataset:
  # The ZIP file you uploaded:
  !unzip dataset.zip
  # Your labels map as a dictionary (zero is reserved):
  label_map = {1: 'apple', 2: 'banana'} 
  if dataset_is_split:
    # If your dataset is already split, specify each path:
    train_images_dir = 'dataset/train/images'
    train_annotations_dir = 'dataset/train/annotations'
    val_images_dir = 'dataset/validation/images'
    val_annotations_dir = 'dataset/validation/annotations'
    test_images_dir = 'dataset/test/images'
    test_annotations_dir = 'dataset/test/annotations'
  else:
    # If it's NOT split yet, specify the path to all images and annotations
    images_in = 'dataset/images'
    annotations_in = 'dataset/annotations'
  
Explanation: If you want to load your own dataset as a CSV file, you can learn more about the format in Formatting a training data CSV. You can load your CSV either from Cloud Storage (as shown above) or from a local path.
DataLoader can also load your dataset in other formats, such as from a set of TFRecord files or from a local directory using the Pascal VOC format (shown below for a custom dataset). 
(Optional) Load your own Pascal VOC dataset
To use your custom dataset, you need to modify a few variables here, such as your ZIP filename, your label map, and the path to your images/annotations:
End of explanation
#@markdown Be sure you run this cell. It's hiding the `split_dataset()` function used in the next code block.
import os
import random
import shutil
def split_dataset(images_path, annotations_path, val_split, test_split, out_path):
  Splits a directory of sorted images/annotations into training, validation, and test sets.
  Args:
    images_path: Path to the directory with your images (JPGs).
    annotations_path: Path to a directory with your VOC XML annotation files,
      with filenames corresponding to image filenames. This may be the same path
      used for images_path.
    val_split: Fraction of data to reserve for validation (float between 0 and 1).
    test_split: Fraction of data to reserve for test (float between 0 and 1).
  Returns:
    The paths for the split images/annotations (train_dir, val_dir, test_dir)
  
  _, dirs, _ = next(os.walk(images_path))
  train_dir = os.path.join(out_path, 'train')
  val_dir = os.path.join(out_path, 'validation')
  test_dir = os.path.join(out_path, 'test')
  IMAGES_TRAIN_DIR = os.path.join(train_dir, 'images')
  IMAGES_VAL_DIR = os.path.join(val_dir, 'images')
  IMAGES_TEST_DIR = os.path.join(test_dir, 'images')
  os.makedirs(IMAGES_TRAIN_DIR, exist_ok=True)
  os.makedirs(IMAGES_VAL_DIR, exist_ok=True)
  os.makedirs(IMAGES_TEST_DIR, exist_ok=True)
  ANNOT_TRAIN_DIR = os.path.join(train_dir, 'annotations')
  ANNOT_VAL_DIR = os.path.join(val_dir, 'annotations')
  ANNOT_TEST_DIR = os.path.join(test_dir, 'annotations')
  os.makedirs(ANNOT_TRAIN_DIR, exist_ok=True)
  os.makedirs(ANNOT_VAL_DIR, exist_ok=True)
  os.makedirs(ANNOT_TEST_DIR, exist_ok=True)
  # Get all filenames for this dir, filtered by filetype
  filenames = os.listdir(os.path.join(images_path))
  filenames = [os.path.join(images_path, f) for f in filenames if (f.endswith('.jpg'))]
  # Shuffle the files, deterministically
  filenames.sort()
  random.seed(42)
  random.shuffle(filenames)
  # Get exact number of images for validation and test; the rest is for training
  val_count = int(len(filenames) * val_split)
  test_count = int(len(filenames) * test_split)
  for i, file in enumerate(filenames):
    source_dir, filename = os.path.split(file)
    annot_file = os.path.join(annotations_path, filename.replace("jpg", "xml"))
    if i < val_count:
      shutil.copy(file, IMAGES_VAL_DIR)
      shutil.copy(annot_file, ANNOT_VAL_DIR)
    elif i < val_count + test_count:
      shutil.copy(file, IMAGES_TEST_DIR)
      shutil.copy(annot_file, ANNOT_TEST_DIR)
    else:
      shutil.copy(file, IMAGES_TRAIN_DIR)
      shutil.copy(annot_file, ANNOT_TRAIN_DIR)
  return (train_dir, val_dir, test_dir)
# We need to instantiate a separate DataLoader for each split dataset
if use_custom_dataset:
  if dataset_is_split:
    train_data = object_detector.DataLoader.from_pascal_voc(
        train_images_dir, train_annotations_dir, label_map=label_map)
    validation_data = object_detector.DataLoader.from_pascal_voc(
        val_images_dir, val_annotations_dir, label_map=label_map)
    test_data = object_detector.DataLoader.from_pascal_voc(
        test_images_dir, test_annotations_dir, label_map=label_map)
  else:
    train_dir, val_dir, test_dir = split_dataset(images_in, annotations_in,
                                                 val_split=0.2, test_split=0.2,
                                                 out_path='split-dataset')
    train_data = object_detector.DataLoader.from_pascal_voc(
        os.path.join(train_dir, 'images'),
        os.path.join(train_dir, 'annotations'), label_map=label_map)
    validation_data = object_detector.DataLoader.from_pascal_voc(
        os.path.join(val_dir, 'images'),
        os.path.join(val_dir, 'annotations'), label_map=label_map)
    test_data = object_detector.DataLoader.from_pascal_voc(
        os.path.join(test_dir, 'images'),
        os.path.join(test_dir, 'annotations'), label_map=label_map)
    
  print(f'train count: {len(train_data)}')
  print(f'validation count: {len(validation_data)}')
  print(f'test count: {len(test_data)}')
Explanation: Now you're ready to train the model with your custom dataset. But before you run the notebook, you should also skip to the Export to TensorFlow Lite section and change the TFLITE_FILENAME and LABLES_FILENAME for your exported files.
Then run the whole notebook by clicking Runtime > Run all.
End of explanation
spec = object_detector.EfficientDetLite0Spec()
Explanation: Select the model spec
Model Maker supports the EfficientDet-Lite family of object detection models that are compatible with the Edge TPU. (EfficientDet-Lite is derived from EfficientDet, which offers state-of-the-art accuracy in a small model size). There are several model sizes you can choose from:
|| Model architecture | Size(MB) | Latency(ms) | Average Precision |
|-|--------------------|-----------|---------------|----------------------|
|| EfficientDet-Lite0 | 5.7       | 37.4            | 30.4%               |
|| EfficientDet-Lite1 | 7.6       | 56.3            | 34.3%               |
|| EfficientDet-Lite2 | 10.2      | 104.6           | 36.0%               |
|| EfficientDet-Lite3 | 14.4      | 107.6           | 39.4%               |
| <td colspan=4><br><i> File size of the compiled Edge TPU models. <br/>** Latency measured on a desktop CPU with a Coral USB Accelerator. <br/> Average Precision is the mAP (mean Average Precision) on the COCO 2017 validation dataset.</i></td> |
Beware that the Lite2 and Lite3 models do not fit onto the Edge TPU's onboard memory, so you'll see even greater latency when using those, due to the cost of fetching data from the host system memory. Maybe this extra latency is okay for your application, but if it's not and you require the precision of the larger models, then you can pipeline the model across multiple Edge TPUs (more about this when we compile the model below).
For this tutorial, we'll use Lite0:
End of explanation
model = object_detector.create(train_data=train_data, 
                               model_spec=spec, 
                               validation_data=validation_data, 
                               epochs=50, 
                               batch_size=10, 
                               train_whole_model=True)
Explanation: The EfficientDetLite0Spec constructor also supports several arguments that specify training options, such as the max number of detections (default is 25 for the TF Lite model) and whether to use Cloud TPUs for training. You can also use the constructor to specify the number of training epochs and the batch size, but you can also specify those in the next step.
Create and train the model
Now we need to create our model according to the model spec, load our dataset into the model, specify training parameters, and begin training. 
Using Model Maker, we accomplished all of that with create():
End of explanation
model.evaluate(test_data)
Explanation: Evaluate the model
Now we'll use the test dataset to evaluate how well the model performs with data it has never seen before.
The evaluate() method provides output in the style of COCO evaluation metrics:
End of explanation
TFLITE_FILENAME = 'efficientdet-lite-salad.tflite'
LABELS_FILENAME = 'salad-labels.txt'
model.export(export_dir='.', tflite_filename=TFLITE_FILENAME, label_filename=LABELS_FILENAME,
             export_format=[ExportFormat.TFLITE, ExportFormat.LABEL])
Explanation: Because the default batch size for EfficientDetLite models is 64, this needs only 1 step to go through all 25 images in the salad test set. You can also specify the batch_size argument when you call evaluate().
Export to TensorFlow Lite
Next, we'll export the model to the TensorFlow Lite format. By default, the export() method performs full integer post-training quantization, which is exactly what we need for compatibility with the Edge TPU. (Model Maker uses the same dataset we gave to our model spec as a representative dataset, which is required for full-int quantization.)
We just need to specify the export directory and format. By default, it exports to TF Lite, but we also want a labels file, so we declare both:
End of explanation
model.evaluate_tflite(TFLITE_FILENAME, test_data)
Explanation: Evaluate the TF Lite model
Exporting the model to TensorFlow Lite can affect the model accuracy, due to the reduced numerical precision from quantization and because the original TensorFlow model uses per-class non-max supression (NMS) for post-processing, while the TF Lite model uses global NMS, which is faster but less accurate.
Therefore you should always evaluate the exported TF Lite model and be sure it still meets your requirements:
End of explanation
import random
# If you're using a custom dataset, we take a random image from the test set:
if use_custom_dataset:
  images_path = test_images_dir if dataset_is_split else os.path.join(test_dir, "images")
  filenames = os.listdir(os.path.join(images_path))
  random_index = random.randint(0,len(filenames)-1)
  INPUT_IMAGE = os.path.join(images_path, filenames[random_index])
else:
  # Download a test salad image
  INPUT_IMAGE = 'salad-test.jpg'
  DOWNLOAD_URL = "https://storage.googleapis.com/cloud-ml-data/img/openimage/3/2520/3916261642_0a504acd60_o.jpg"
  !wget -q -O $INPUT_IMAGE $DOWNLOAD_URL
Explanation: Try the TFLite model
Just to be sure of things, let's run the model ourselves with an image from the test set.
End of explanation
! python3 -m pip install --extra-index-url https://google-coral.github.io/py-repo/ pycoral
from PIL import Image
from PIL import ImageDraw
from PIL import ImageFont
import tflite_runtime.interpreter as tflite 
from pycoral.adapters import common
from pycoral.adapters import detect
from pycoral.utils.dataset import read_label_file
def draw_objects(draw, objs, scale_factor, labels):
  Draws the bounding box and label for each object.
  COLORS = np.random.randint(0, 255, size=(len(labels), 3), dtype=np.uint8)
  for obj in objs:
    bbox = obj.bbox
    color = tuple(int(c) for c in COLORS[obj.id])
    draw.rectangle([(bbox.xmin * scale_factor, bbox.ymin * scale_factor),
                    (bbox.xmax * scale_factor, bbox.ymax * scale_factor)],
                   outline=color, width=3)
    font = ImageFont.truetype("LiberationSans-Regular.ttf", size=15)
    draw.text((bbox.xmin * scale_factor + 4, bbox.ymin * scale_factor + 4),
              '%s\n%.2f' % (labels.get(obj.id, obj.id), obj.score),
              fill=color, font=font)
# Load the TF Lite model
labels = read_label_file(LABELS_FILENAME)
interpreter = tflite.Interpreter(TFLITE_FILENAME)
interpreter.allocate_tensors()
# Resize the image for input
image = Image.open(INPUT_IMAGE)
_, scale = common.set_resized_input(
    interpreter, image.size, lambda size: image.resize(size, Image.ANTIALIAS))
# Run inference
interpreter.invoke()
objs = detect.get_objects(interpreter, score_threshold=0.4, image_scale=scale)
# Resize again to a reasonable size for display
display_width = 500
scale_factor = display_width / image.width
height_ratio = image.height / image.width
image = image.resize((display_width, int(display_width * height_ratio)))
draw_objects(ImageDraw.Draw(image), objs, scale_factor, labels)
image
Explanation: To simplify our code, we'll use the PyCoral API:
End of explanation
! curl https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add -
! echo "deb https://packages.cloud.google.com/apt coral-edgetpu-stable main" | sudo tee /etc/apt/sources.list.d/coral-edgetpu.list
! sudo apt-get update
! sudo apt-get install edgetpu-compiler
Explanation: Compile for the Edge TPU
First we need to download the Edge TPU Compiler:
End of explanation
NUMBER_OF_TPUS =  1
!edgetpu_compiler $TFLITE_FILENAME -d --num_segments=$NUMBER_OF_TPUS
Explanation: Before compiling the .tflite file for the Edge TPU, it's important to consider whether your model will fit into the Edge TPU memory. 
The Edge TPU has approximately 8 MB of SRAM for caching model paramaters, so any model close to or over 8 MB will not fit onto the Edge TPU memory. That means the inference times are longer, because some model parameters must be fetched from the host system memory.
One way to elimiate the extra latency is to use model pipelining, which splits the model into segments that can run on separate Edge TPUs in series. This can significantly reduce the latency for big models.
The following table provides recommendations for the number of Edge TPUs to use with each EfficientDet-Lite model.
| Model architecture | Minimum TPUs | Recommended TPUs
|--------------------|-------|-------|
| EfficientDet-Lite0 | 1     | 1     |
| EfficientDet-Lite1 | 1     | 1     |
| EfficientDet-Lite2 | 1     | 2     |
| EfficientDet-Lite3 | 2     | 2     |
| EfficientDet-Lite4 | 2     | 3     |
If you need extra Edge TPUs for your model, then update NUMBER_OF_TPUS here:
End of explanation
from google.colab import files
files.download(TFLITE_FILENAME)
files.download(TFLITE_FILENAME.replace('.tflite', '_edgetpu.tflite'))
files.download(LABELS_FILENAME)
Explanation: Beware when using multiple segments: The Edge TPU Comiler divides the model such that all segments have roughly equal amounts of parameter data, but that does not mean all segments have the same latency. Especially when dividing an SSD model such as EfficientDet, this results in a latency-imbalance between segments, because SSD models have a large post-processing op that actually executes on the CPU, not on the Edge TPU. So although segmenting your model this way is better than running the whole model on just one Edge TPU, we recommend that you segment the EfficientDet-Lite model using our profiling-based partitioner tool, which measures each segment's latency on the Edge TPU and then iteratively adjusts the segmentation sizes to provide balanced latency between all segments.
Download the files
End of explanation
<END_TASK>