<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 Generating batches
In this example we will explore how to read a simple Lightning Memory-Mapped Database (LMDB) with pyxis using iterators.
Step1:  For consistency, we will be using a random number generator with a seed for some of the iterators.
Step2:  Let's start by creating a small dataset of 10 samples. Each input is a randomly generated image with shape (254, 254, 3), while the targets are scalar values.
Step3:  The data is written using the pyxis writer.
Step4:  Using batch iterators
Read back the data using the pyxis reader.
Step5:  Example 1 - Number of samples is a multiple of the batch size
In this first example we create a (simple) batch iterator where the number of samples is divisible by the batch size.
Step6:  All the iterators that come with pyxis have the mandatory keys argument. The data returned by the iterator will be the values for which these keys point to. The order of the keys matter. For example, when using the keys ('a', 'b') the iterator will return (a_val, b_val), where a_val and b_val are the values associated with the keys 'a' and 'b', respectively.
The artificial dataset has 10 samples, so by letting the batch size be 5 it will take two iterations to go through the whole dataset. The artificial targets for four batches are printed out to showcase this.
endless is by default on, which means that after having gone through the dataset, the iterator will re-iterate over the data.
Step7:  Example 2 - Number of samples is not a multiple of the batch size
Step8:  The artificial dataset has 10 samples, so by letting the batch size be 3 it will take four iterations to go through the whole dataset. The artificial targets for six batches are printed out to showcase this.
Notice that the final batch of the dataset only contains the remaining unseen samples.
Step9:  Example 3 - Shuffling of data
Until now we have created batches by reading samples from the dataset in the order they were written. However, by turning shuffling on, the samples in the dataset will be reshuffled each time we go through the dataset.
Notice how we only request the values for the y key this time.
Step10:  Example 4 - Stochastic batch iterator
Batches can be created stochastically. This means that the samples in a batch are sampled uniformly from the entire dataset. Here we showcase ten different batches with a batch size of five.
Step11:  Example 5 - Sequential batch iterator
Batches can be created by reading the database sequentially. This means that the samples in a batch are not shuffled, but can be read at a higher speed. The sequential batch iterator is ideal for very large datasets. Here we showcase ten different batches with a batch size of 3.
Step12:  Example 6 - Thread-safe iterators
The three types of iterators demonstrated so far are
Step13:  SquareTargets can now be used to generate batches of data from the LMDB.
Step14:  Close everything
We should make sure to close the LMDB environment after we are done reading.
<ASSISTANT_TASK:>
Python Code:
from __future__ import print_function
import numpy as np
import pyxis as px
Explanation: Generating batches
In this example we will explore how to read a simple Lightning Memory-Mapped Database (LMDB) with pyxis using iterators.
End of explanation
rng = np.random.RandomState(1234)
Explanation: For consistency, we will be using a random number generator with a seed for some of the iterators.
End of explanation
nb_samples = 10
X = rng.rand(nb_samples, 254, 254, 3)
y = np.arange(nb_samples, dtype=np.uint8)
Explanation: Let's start by creating a small dataset of 10 samples. Each input is a randomly generated image with shape (254, 254, 3), while the targets are scalar values.
End of explanation
db = px.Writer(dirpath='data', map_size_limit=30, ram_gb_limit=1)
db.put_samples('X', X, 'y', y)
db.close()
Explanation: The data is written using the pyxis writer.
End of explanation
db = px.Reader('data')
Explanation: Using batch iterators
Read back the data using the pyxis reader.
End of explanation
gen = px.SimpleBatch(db, keys=('X', 'y'), batch_size=5, shuffle=False)
Explanation: Example 1 - Number of samples is a multiple of the batch size
In this first example we create a (simple) batch iterator where the number of samples is divisible by the batch size.
End of explanation
for i in range(4):
    xs, ys = next(gen)
    print()
    print('Iteration:', i, '\tTargets:', ys)
    if gen.end_of_dataset:
        print('We have reached the end of the dataset')
Explanation: All the iterators that come with pyxis have the mandatory keys argument. The data returned by the iterator will be the values for which these keys point to. The order of the keys matter. For example, when using the keys ('a', 'b') the iterator will return (a_val, b_val), where a_val and b_val are the values associated with the keys 'a' and 'b', respectively.
The artificial dataset has 10 samples, so by letting the batch size be 5 it will take two iterations to go through the whole dataset. The artificial targets for four batches are printed out to showcase this.
endless is by default on, which means that after having gone through the dataset, the iterator will re-iterate over the data.
End of explanation
gen = px.SimpleBatch(db, keys=('X', 'y'), batch_size=3, shuffle=False)
Explanation: Example 2 - Number of samples is not a multiple of the batch size
End of explanation
for i in range(6):
    xs, ys = next(gen)
    print()
    print('Iteration:', i, '\tTargets:', ys)
    if gen.end_of_dataset:
        print('We have reached the end of the dataset')
Explanation: The artificial dataset has 10 samples, so by letting the batch size be 3 it will take four iterations to go through the whole dataset. The artificial targets for six batches are printed out to showcase this.
Notice that the final batch of the dataset only contains the remaining unseen samples.
End of explanation
gen = px.SimpleBatch(db, keys=('y'), batch_size=5, shuffle=True, rng=rng)
for i in range(6):
    ys = next(gen)
    print()
    print('Iteration:', i, '\tTargets:', ys)
    if gen.end_of_dataset:
        print('We have reached the end of the dataset')
Explanation: Example 3 - Shuffling of data
Until now we have created batches by reading samples from the dataset in the order they were written. However, by turning shuffling on, the samples in the dataset will be reshuffled each time we go through the dataset.
Notice how we only request the values for the y key this time.
End of explanation
gen = px.StochasticBatch(db, keys=('y'), batch_size=5, rng=rng)
for i in range(10):
    ys = next(gen)
    print('Iteration:', i, '\tTargets:', ys)
Explanation: Example 4 - Stochastic batch iterator
Batches can be created stochastically. This means that the samples in a batch are sampled uniformly from the entire dataset. Here we showcase ten different batches with a batch size of five.
End of explanation
gen = px.SequentialBatch(db, keys=('y'), batch_size=3)
for i in range(10):
    ys = next(gen)
    print('Iteration:', i, '\tTargets:', ys)
Explanation: Example 5 - Sequential batch iterator
Batches can be created by reading the database sequentially. This means that the samples in a batch are not shuffled, but can be read at a higher speed. The sequential batch iterator is ideal for very large datasets. Here we showcase ten different batches with a batch size of 3.
End of explanation
class SquareTargets(px.SimpleBatchThreadSafe):
    def __init__(self, db, keys, batch_size):
        super(SquareTargets, self).__init__(db, keys, batch_size,
                                            shuffle=False,
                                            endless=False)
    def __next__(self):
        with self.lock:
            X, y = next(self.gen)
        y = y ** 2
        return X, y
Explanation: Example 6 - Thread-safe iterators
The three types of iterators demonstrated so far are:
pyxis.SimpleBatch
pyxis.StochasticBatch
pyxis.SequentialBatch
Each of these come with a thread-safe variant. By thread-safe we mean that when more than one thread make use of the iterator it will not raise an exception. These variants have the suffix ThreadSafe:
pyxis.SimpleBatchThreadSafe
pyxis.StochasticBatchThreadSafe
pyxis.SequentialBatchThreadSafe
Other than being thread-safe, they work exactly the same as the non-thread-safe versions.
Custom iterators
The output of the types of iterators demonstrated above will always yield data as they were stored in the LMDB.
To create an iterator that modifies the data we can, for example, modify one of the existing iterators using inheritance. Here is an example where all targets are squared before they are output by the iterator. Notice how thread-safety is achieved by using the with statement with the Python lock object.
End of explanation
gen = SquareTargets(db, keys=('X', 'y'), batch_size=2)
print('Squared targets:')
for _, y in gen:
    print(y)
Explanation: SquareTargets can now be used to generate batches of data from the LMDB.
End of explanation
db.close()
Explanation: Close everything
We should make sure to close the LMDB environment after we are done reading.
End of explanation
<END_TASK>