<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 TensorFlow Datasets
TFDS provides a collection of ready-to-use datasets for use with TensorFlow, Jax, and other Machine Learning frameworks.
It handles downloading and preparing the data deterministically and constructing a tf.data.Dataset (or np.array).
Note
Step1:  Find available datasets
All dataset builders are subclass of tfds.core.DatasetBuilder. To get the list of available builders, use tfds.list_builders() or look at our catalog.
Step2:  Load a dataset
tfds.load
The easiest way of loading a dataset is tfds.load. It will
Step3:  Some common arguments
Step4:  tfds build CLI
If you want to generate a specific dataset, you can use the tfds command line. For example
Step5:  To find out the dict key names and structure, look at the dataset documentation in our catalog. For example
Step6:  As numpy (tfds.as_numpy)
Uses tfds.as_numpy to convert
Step7:  As batched tf.Tensor (batch_size=-1)
By using batch_size=-1, you can load the full dataset in a single batch.
This can be combined with as_supervised=True and tfds.as_numpy to get the the data as (np.array, np.array)
Step8:  Be careful that your dataset can fit in memory, and that all examples have the same shape.
Benchmark your datasets
Benchmarking a dataset is a simple tfds.benchmark call on any iterable (e.g. tf.data.Dataset, tfds.as_numpy,...).
Step9:  Do not forget to normalize the results per batch size with the batch_size= kwarg.
In the summary, the first warmup batch is separated from the other ones to capture tf.data.Dataset extra setup time (e.g. buffers initialization,...).
Notice how the second iteration is much faster due to TFDS auto-caching.
tfds.benchmark returns a tfds.core.BenchmarkResult which can be inspected for further analysis.
Build end-to-end pipeline
To go further, you can look
Step10:  tfds.show_examples
tfds.show_examples returns a matplotlib.figure.Figure (only image datasets supported now)
Step11:  Access the dataset metadata
All builders include a tfds.core.DatasetInfo object containing the dataset metadata.
It can be accessed through
Step12:  The tfds.core.DatasetBuilder API
Step13:  The dataset info contains additional informations about the dataset (version, citation, homepage, description,...).
Step14:  Features metadata (label names, image shape,...)
Access the tfds.features.FeatureDict
Step15:  Number of classes, label names
Step16:  Shapes, dtypes
Step17:  Split metadata (e.g. split names, number of examples,...)
Access the tfds.core.SplitDict
Step18:  Available splits
Step19:  Get info on individual split
Step20:  It also works with the subsplit API
<ASSISTANT_TASK:>
Python Code:
!pip install -q tfds-nightly tensorflow matplotlib
import matplotlib.pyplot as plt
import numpy as np
import tensorflow as tf
import tensorflow_datasets as tfds
Explanation: TensorFlow Datasets
TFDS provides a collection of ready-to-use datasets for use with TensorFlow, Jax, and other Machine Learning frameworks.
It handles downloading and preparing the data deterministically and constructing a tf.data.Dataset (or np.array).
Note: Do not confuse TFDS (this library) with tf.data (TensorFlow API to build efficient data pipelines). TFDS is a high level wrapper around tf.data. If you're not familiar with this API, we encourage you to read the official tf.data guide first.
Copyright 2018 The TensorFlow Datasets Authors, Licensed under the Apache License, Version 2.0
<table class="tfo-notebook-buttons" align="left">
  <td>
    <a target="_blank" href="https://www.tensorflow.org/datasets/overview"><img src="https://www.tensorflow.org/images/tf_logo_32px.png" />View on TensorFlow.org</a>
  </td>
  <td>
    <a target="_blank" href="https://colab.research.google.com/github/tensorflow/datasets/blob/master/docs/overview.ipynb"><img src="https://www.tensorflow.org/images/colab_logo_32px.png" />Run in Google Colab</a>
  </td>
  <td>
    <a target="_blank" href="https://github.com/tensorflow/datasets/blob/master/docs/overview.ipynb"><img src="https://www.tensorflow.org/images/GitHub-Mark-32px.png" />View source on GitHub</a>
  </td>
  <td>
    <a href="https://storage.googleapis.com/tensorflow_docs/datasets/docs/overview.ipynb"><img src="https://www.tensorflow.org/images/download_logo_32px.png" />Download notebook</a>
  </td>
</table>
Installation
TFDS exists in two packages:
pip install tensorflow-datasets: The stable version, released every few months.
pip install tfds-nightly: Released every day, contains the last versions of the datasets.
This colab uses tfds-nightly:
End of explanation
tfds.list_builders()
Explanation: Find available datasets
All dataset builders are subclass of tfds.core.DatasetBuilder. To get the list of available builders, use tfds.list_builders() or look at our catalog.
End of explanation
ds = tfds.load('mnist', split='train', shuffle_files=True)
assert isinstance(ds, tf.data.Dataset)
print(ds)
Explanation: Load a dataset
tfds.load
The easiest way of loading a dataset is tfds.load. It will:
Download the data and save it as tfrecord files.
Load the tfrecord and create the tf.data.Dataset.
End of explanation
builder = tfds.builder('mnist')
# 1. Create the tfrecord files (no-op if already exists)
builder.download_and_prepare()
# 2. Load the `tf.data.Dataset`
ds = builder.as_dataset(split='train', shuffle_files=True)
print(ds)
Explanation: Some common arguments:
split=: Which split to read (e.g. 'train', ['train', 'test'], 'train[80%:]',...). See our split API guide.
shuffle_files=: Control whether to shuffle the files between each epoch (TFDS store big datasets in multiple smaller files).
data_dir=: Location where the dataset is saved (
defaults to ~/tensorflow_datasets/)
with_info=True: Returns the tfds.core.DatasetInfo containing dataset metadata
download=False: Disable download
tfds.builder
tfds.load is a thin wrapper around tfds.core.DatasetBuilder. You can get the same output using the tfds.core.DatasetBuilder API:
End of explanation
ds = tfds.load('mnist', split='train')
ds = ds.take(1)  # Only take a single example
for example in ds:  # example is `{'image': tf.Tensor, 'label': tf.Tensor}`
  print(list(example.keys()))
  image = example["image"]
  label = example["label"]
  print(image.shape, label)
Explanation: tfds build CLI
If you want to generate a specific dataset, you can use the tfds command line. For example:
sh
tfds build mnist
See the doc for available flags.
Iterate over a dataset
As dict
By default, the tf.data.Dataset object contains a dict of tf.Tensors:
End of explanation
ds = tfds.load('mnist', split='train', as_supervised=True)
ds = ds.take(1)
for image, label in ds:  # example is (image, label)
  print(image.shape, label)
Explanation: To find out the dict key names and structure, look at the dataset documentation in our catalog. For example: mnist documentation.
As tuple (as_supervised=True)
By using as_supervised=True, you can get a tuple (features, label) instead for supervised datasets.
End of explanation
ds = tfds.load('mnist', split='train', as_supervised=True)
ds = ds.take(1)
for image, label in tfds.as_numpy(ds):
  print(type(image), type(label), label)
Explanation: As numpy (tfds.as_numpy)
Uses tfds.as_numpy to convert:
tf.Tensor -> np.array
tf.data.Dataset -> Iterator[Tree[np.array]] (Tree can be arbitrary nested Dict, Tuple)
End of explanation
image, label = tfds.as_numpy(tfds.load(
    'mnist',
    split='test',
    batch_size=-1,
    as_supervised=True,
))
print(type(image), image.shape)
Explanation: As batched tf.Tensor (batch_size=-1)
By using batch_size=-1, you can load the full dataset in a single batch.
This can be combined with as_supervised=True and tfds.as_numpy to get the the data as (np.array, np.array):
End of explanation
ds = tfds.load('mnist', split='train')
ds = ds.batch(32).prefetch(1)
tfds.benchmark(ds, batch_size=32)
tfds.benchmark(ds, batch_size=32)  # Second epoch much faster due to auto-caching
Explanation: Be careful that your dataset can fit in memory, and that all examples have the same shape.
Benchmark your datasets
Benchmarking a dataset is a simple tfds.benchmark call on any iterable (e.g. tf.data.Dataset, tfds.as_numpy,...).
End of explanation
ds, info = tfds.load('mnist', split='train', with_info=True)
tfds.as_dataframe(ds.take(4), info)
Explanation: Do not forget to normalize the results per batch size with the batch_size= kwarg.
In the summary, the first warmup batch is separated from the other ones to capture tf.data.Dataset extra setup time (e.g. buffers initialization,...).
Notice how the second iteration is much faster due to TFDS auto-caching.
tfds.benchmark returns a tfds.core.BenchmarkResult which can be inspected for further analysis.
Build end-to-end pipeline
To go further, you can look:
Our end-to-end Keras example to see a full training pipeline (with batching, shuffling,...).
Our performance guide to improve the speed of your pipelines (tip: use tfds.benchmark(ds) to benchmark your datasets).
Visualization
tfds.as_dataframe
tf.data.Dataset objects can be converted to pandas.DataFrame with tfds.as_dataframe to be visualized on Colab.
Add the tfds.core.DatasetInfo as second argument of tfds.as_dataframe to visualize images, audio, texts, videos,...
Use ds.take(x) to only display the first x examples. pandas.DataFrame will load the full dataset in-memory, and can be very expensive to display.
End of explanation
ds, info = tfds.load('mnist', split='train', with_info=True)
fig = tfds.show_examples(ds, info)
Explanation: tfds.show_examples
tfds.show_examples returns a matplotlib.figure.Figure (only image datasets supported now):
End of explanation
ds, info = tfds.load('mnist', with_info=True)
Explanation: Access the dataset metadata
All builders include a tfds.core.DatasetInfo object containing the dataset metadata.
It can be accessed through:
The tfds.load API:
End of explanation
builder = tfds.builder('mnist')
info = builder.info
Explanation: The tfds.core.DatasetBuilder API:
End of explanation
print(info)
Explanation: The dataset info contains additional informations about the dataset (version, citation, homepage, description,...).
End of explanation
info.features
Explanation: Features metadata (label names, image shape,...)
Access the tfds.features.FeatureDict:
End of explanation
print(info.features["label"].num_classes)
print(info.features["label"].names)
print(info.features["label"].int2str(7))  # Human readable version (8 -> 'cat')
print(info.features["label"].str2int('7'))
Explanation: Number of classes, label names:
End of explanation
print(info.features.shape)
print(info.features.dtype)
print(info.features['image'].shape)
print(info.features['image'].dtype)
Explanation: Shapes, dtypes:
End of explanation
print(info.splits)
Explanation: Split metadata (e.g. split names, number of examples,...)
Access the tfds.core.SplitDict:
End of explanation
print(list(info.splits.keys()))
Explanation: Available splits:
End of explanation
print(info.splits['train'].num_examples)
print(info.splits['train'].filenames)
print(info.splits['train'].num_shards)
Explanation: Get info on individual split:
End of explanation
print(info.splits['train[15%:75%]'].num_examples)
print(info.splits['train[15%:75%]'].file_instructions)
Explanation: It also works with the subsplit API:
End of explanation
<END_TASK>