<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 Working With Datasets
Data is central to machine learning.  This tutorial introduces the Dataset class that DeepChem uses to store and manage data.  It provides simple but powerful tools for efficiently working with large amounts of data.  It also is designed to easily interact with other popular Python frameworks such as NumPy, Pandas, TensorFlow, and PyTorch.
Colab
This tutorial and the rest in this sequence can be done in Google colab. If you'd like to open this notebook in colab, you can use the following link.
Step1:  We can now import the deepchem package to play with.
Step2:  Anatomy of a Dataset
In the last tutorial we loaded the Delaney dataset of molecular solubilities.  Let's load it again.
Step3:  We now have three Dataset objects
Step4:  There's a lot of information there, so let's start at the beginning.  It begins with the label "DiskDataset".  Dataset is an abstract class.  It has a few subclasses that correspond to different ways of storing data.
DiskDataset is a dataset that has been saved to disk.  The data is stored in a way that can be efficiently accessed, even if the total amount of data is far larger than your computer's memory.
NumpyDataset is an in-memory dataset that holds all the data in NumPy arrays.  It is a useful tool when manipulating small to medium sized datasets that can fit entirely in memory.
ImageDataset is a more specialized class that stores some or all of the data in image files on disk.  It is useful when working with models that have images as their inputs or outputs.
Now let's consider the contents of the Dataset.  Every Dataset stores a list of samples.  Very roughly speaking, a sample is a single data point.  In this case, each sample is a molecule.  In other datasets a sample might correspond to an experimental assay, a cell line, an image, or many other things.  For every sample the dataset stores the following information.
The features, referred to as X.  This is the input that should be fed into a model to represent the sample.
The labels, referred to as y.  This is the desired output from the model.  During training, it tries to make the model's output for each sample as close as possible to y.
The weights, referred to as w.  This can be used to indicate that some data values are more important than others.  In later tutorials we will see examples of how this is useful.
An ID, which is a unique identifier for the sample.  This can be anything as long as it is unique.  Sometimes it is just an integer index, but in this dataset the ID is a SMILES string describing the molecule.
Notice that X, y, and w all have 113 as the size of their first dimension.  That means this dataset contains 113 samples.
The final piece of information listed in the output is task_names.  Some datasets contain multiple pieces of information for each sample.  For example, if a sample represents a molecule, the dataset might record the results of several different experiments on that molecule.  This dataset has only a single task
Step5:  This is a very easy way to access data, but you should be very careful about using it.  This requires the data for all samples to be loaded into memory at once.  That's fine for small datasets like this one, but for large datasets it could easily take more memory than you have.
A better approach is to iterate over the dataset.  That lets it load just a little data at a time, process it, then free the memory before loading the next bit.  You can use the itersamples() method to iterate over samples one at a time.
Step6:  Most deep learning models can process a batch of multiple samples all at once.  You can use iterbatches() to iterate over batches of samples.
Step7:  iterbatches() has other features that are useful when training models.  For example, iterbatches(batch_size=100, epochs=10, deterministic=False) will iterate over the complete dataset ten times, each time with the samples in a different random order.
Datasets can also expose data using the standard interfaces for TensorFlow and PyTorch.  To get a tensorflow.data.Dataset, call make_tf_dataset().  To get a torch.utils.data.IterableDataset, call make_pytorch_dataset().  See the API documentation for more details.
The final way of accessing data is to_dataframe().  This copies the data into a Pandas DataFrame.  This requires storing all the data in memory at once, so you should only use it with small datasets.
Step8:  Creating Datasets
Now let's talk about how you can create your own datasets.  Creating a NumpyDataset is very simple
Step9:  Notice that we did not specify weights or IDs.  These are optional, as is y for that matter.  Only X is required.  Since we left them out, it automatically built w and ids arrays for us, setting all weights to 1 and setting the IDs to integer indices.
Step10:  What about creating a DiskDataset?  If you have the data in NumPy arrays, you can call DiskDataset.from_numpy() to save it to disk.  Since this is just a tutorial, we will save it to a temporary directory.
<ASSISTANT_TASK:>
Python Code:
!pip install --pre deepchem
Explanation: Working With Datasets
Data is central to machine learning.  This tutorial introduces the Dataset class that DeepChem uses to store and manage data.  It provides simple but powerful tools for efficiently working with large amounts of data.  It also is designed to easily interact with other popular Python frameworks such as NumPy, Pandas, TensorFlow, and PyTorch.
Colab
This tutorial and the rest in this sequence can be done in Google colab. If you'd like to open this notebook in colab, you can use the following link.
End of explanation
import deepchem as dc
dc.__version__
Explanation: We can now import the deepchem package to play with.
End of explanation
tasks, datasets, transformers = dc.molnet.load_delaney(featurizer='GraphConv')
train_dataset, valid_dataset, test_dataset = datasets
Explanation: Anatomy of a Dataset
In the last tutorial we loaded the Delaney dataset of molecular solubilities.  Let's load it again.
End of explanation
print(test_dataset)
Explanation: We now have three Dataset objects: the training, validation, and test sets.  What information does each of them contain?  We can start to get an idea by printing out the string representation of one of them.
End of explanation
test_dataset.y
Explanation: There's a lot of information there, so let's start at the beginning.  It begins with the label "DiskDataset".  Dataset is an abstract class.  It has a few subclasses that correspond to different ways of storing data.
DiskDataset is a dataset that has been saved to disk.  The data is stored in a way that can be efficiently accessed, even if the total amount of data is far larger than your computer's memory.
NumpyDataset is an in-memory dataset that holds all the data in NumPy arrays.  It is a useful tool when manipulating small to medium sized datasets that can fit entirely in memory.
ImageDataset is a more specialized class that stores some or all of the data in image files on disk.  It is useful when working with models that have images as their inputs or outputs.
Now let's consider the contents of the Dataset.  Every Dataset stores a list of samples.  Very roughly speaking, a sample is a single data point.  In this case, each sample is a molecule.  In other datasets a sample might correspond to an experimental assay, a cell line, an image, or many other things.  For every sample the dataset stores the following information.
The features, referred to as X.  This is the input that should be fed into a model to represent the sample.
The labels, referred to as y.  This is the desired output from the model.  During training, it tries to make the model's output for each sample as close as possible to y.
The weights, referred to as w.  This can be used to indicate that some data values are more important than others.  In later tutorials we will see examples of how this is useful.
An ID, which is a unique identifier for the sample.  This can be anything as long as it is unique.  Sometimes it is just an integer index, but in this dataset the ID is a SMILES string describing the molecule.
Notice that X, y, and w all have 113 as the size of their first dimension.  That means this dataset contains 113 samples.
The final piece of information listed in the output is task_names.  Some datasets contain multiple pieces of information for each sample.  For example, if a sample represents a molecule, the dataset might record the results of several different experiments on that molecule.  This dataset has only a single task: "measured log solubility in mols per litre".  Also notice that y and w each have shape (113, 1).  The second dimension of these arrays usually matches the number of tasks.
Accessing Data from a Dataset
There are many ways to access the data contained in a dataset.  The simplest is just to directly access the X, y, w, and ids properties.  Each of these returns the corresponding information as a NumPy array.
End of explanation
for X, y, w, id in test_dataset.itersamples():
    print(y, id)
Explanation: This is a very easy way to access data, but you should be very careful about using it.  This requires the data for all samples to be loaded into memory at once.  That's fine for small datasets like this one, but for large datasets it could easily take more memory than you have.
A better approach is to iterate over the dataset.  That lets it load just a little data at a time, process it, then free the memory before loading the next bit.  You can use the itersamples() method to iterate over samples one at a time.
End of explanation
for X, y, w, ids in test_dataset.iterbatches(batch_size=50):
    print(y.shape)
Explanation: Most deep learning models can process a batch of multiple samples all at once.  You can use iterbatches() to iterate over batches of samples.
End of explanation
test_dataset.to_dataframe()
Explanation: iterbatches() has other features that are useful when training models.  For example, iterbatches(batch_size=100, epochs=10, deterministic=False) will iterate over the complete dataset ten times, each time with the samples in a different random order.
Datasets can also expose data using the standard interfaces for TensorFlow and PyTorch.  To get a tensorflow.data.Dataset, call make_tf_dataset().  To get a torch.utils.data.IterableDataset, call make_pytorch_dataset().  See the API documentation for more details.
The final way of accessing data is to_dataframe().  This copies the data into a Pandas DataFrame.  This requires storing all the data in memory at once, so you should only use it with small datasets.
End of explanation
import numpy as np
X = np.random.random((10, 5))
y = np.random.random((10, 2))
dataset = dc.data.NumpyDataset(X=X, y=y)
print(dataset)
Explanation: Creating Datasets
Now let's talk about how you can create your own datasets.  Creating a NumpyDataset is very simple: just pass the arrays containing the data to the constructor.  Let's create some random arrays, then wrap them in a NumpyDataset.
End of explanation
dataset.to_dataframe()
Explanation: Notice that we did not specify weights or IDs.  These are optional, as is y for that matter.  Only X is required.  Since we left them out, it automatically built w and ids arrays for us, setting all weights to 1 and setting the IDs to integer indices.
End of explanation
import tempfile
with tempfile.TemporaryDirectory() as data_dir:
    disk_dataset = dc.data.DiskDataset.from_numpy(X=X, y=y, data_dir=data_dir)
    print(disk_dataset)
Explanation: What about creating a DiskDataset?  If you have the data in NumPy arrays, you can call DiskDataset.from_numpy() to save it to disk.  Since this is just a tutorial, we will save it to a temporary directory.
End of explanation
<END_TASK>