<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 Input formats
Data in Thunder can be loaded from a variety of formats and locations. We support several input formats for both image and series data, but also encourage a set of standard, recommended formats for these data, especially when working with large datasets.
Setup plotting
Step1:  Loading Images
Images data are distributed collections of images or volumes. Two-dimensional images can be loaded from png or tif files, and three-dimensional volumes can be loaded from multi-page tif stacks, or flat binary volumes. For an example, we'll load a set of tif images by specifying a folder (note
Step2:  Now load the images
Step3:  To look at the first image
Step4:  To check the dimensions of the images
Step5:  And to check the total number of images
Step6:  For any of these image formats, the images can be stored on a local file system (for local use), a networked file system (accessible to all nodes of a cluster), Amazon S3, or Google Storage. To load images from S3, the location of the data must be specified as a URI, with "s3
Step7:  Flat binary files provide a particularly simple format for image data. We recommended storing one image per binary file, alongside a single file conf.json specifying the dimensions and the numerical type
Step8:  We can now load this array directly as an Images object
Step9:  When loading n-dimensional array data, the first dimension is assumed to index the images, so a 4D array will be treated as a collection of 3D volumes, and a 3D array will be treated as a collection of 2D images.
Step10:  Loading Series
A Series object is a distributed collection of one-dimensional arrays with tuple key identifiers. All arrays in a Series must have the same length. They can be loaded from flat text or binary files.
Text files must contain a line for each record, with numbers separated by spaces. The first numbers of each line will be interpreted as keys, and subsequent numbers will be interpreted as values. The number of keys is user-specified. As before, we'll load example series data from a small file included with Thunder purely for testing purposes (this is the iris dataset).
Step11:  If data were split across multiple files, we could have also provided the folder name, rather than the file, and all file(s) of the given format would be loaded. Let's look at the first entry of the Series
Step12:  The index is automatically calculated based on the length of the values array
Step13:  For comparison, you can look at the first raw line of the text file
Step14:  Flat binary files must store each record as a contiguous sequence of bytes, with a fixed size in bytes for each record, including both keys and values. The number and numerical type of keys and records are most convieniently specified in a configuration file in the same directory as the data, but can also be specified as input arguments. Here, we show two ways of loading a binary version of the same iris data loaded previously.
Step15:  In this case, the number of keys and number of values in each record, along with the data types of the keys and values, are automatically read out from a conf.json file located in the same directory as iris.bin. This file has the following simple JSON format.
{
  "keytype"
Step16:  Flat binary files are a particularly convienient format when exporting large data sets from other scientific computing environments, e.g. Matlab. For example, the data loaded above was written out from within Matlab using
f = fopen('iris.bin','w')
fwrite(f, [[0
Step17:  Finally, like Images, a Series object can be constructed directly from a numpy array in memory. To see this, we'll collect the data we just loaded as an array, and then use it to create a new Series object
<ASSISTANT_TASK:>
Python Code:
%matplotlib inline
import matplotlib.pyplot as plt
import seaborn as sns
sns.set_context('notebook')
from thunder import Colorize
image = Colorize.image
Explanation: Input formats
Data in Thunder can be loaded from a variety of formats and locations. We support several input formats for both image and series data, but also encourage a set of standard, recommended formats for these data, especially when working with large datasets.
Setup plotting
End of explanation
import os.path as pth
imagepath = pth.join(pth.dirname(pth.realpath(thunder.__file__)), 'utils/data/fish/images')
Explanation: Loading Images
Images data are distributed collections of images or volumes. Two-dimensional images can be loaded from png or tif files, and three-dimensional volumes can be loaded from multi-page tif stacks, or flat binary volumes. For an example, we'll load a set of tif images by specifying a folder (note: these are small, highly downsampled images included with Thunder purely for demonstration and testing):
For these examples, first we find the location relative to the thunder installation
End of explanation
data = tsc.loadImages(imagepath, inputFormat='tif')
Explanation: Now load the images
End of explanation
img = data.values().first()
image(img[:,:,0])
Explanation: To look at the first image:
End of explanation
data.dims.count
Explanation: To check the dimensions of the images:
End of explanation
data.nrecords
Explanation: And to check the total number of images:
End of explanation
data = tsc.loadImages(imagepath, inputFormat='tif', startIdx=0, stopIdx=10)
data.nrecords
Explanation: For any of these image formats, the images can be stored on a local file system (for local use), a networked file system (accessible to all nodes of a cluster), Amazon S3, or Google Storage. To load images from S3, the location of the data must be specified as a URI, with "s3://" or "s3n://" given as the scheme.
 For instance, data stored in an S3 bucket named "my-bucket" under keys named "my-data/images0.tif, my-data/images1.tif, ..." could be retrieved by passing 's3n://my-bucket/my-data/images*.tif' to loadImages.
It's also easy to load only a subset of images (indexed assuming alphanumeric ordering):
End of explanation
data = tsc.loadImages(imagepath, inputFormat='tif', startIdx=0, stopIdx=10)
arrys = data.collectValuesAsArray()
arrys.shape
Explanation: Flat binary files provide a particularly simple format for image data. We recommended storing one image per binary file, alongside a single file conf.json specifying the dimensions and the numerical type:
{
  "dims": [64, 64], 
  "dtype": "int16"
}
If this file is included in the folder with the binary files it will be used automatically, but the parameters can also be passed as arguments to loadImages
You can load an Images object directly from a numpy arrays in memory. To see that, first we'll collect the images  from the tif files into a local array.
End of explanation
datanew = tsc.loadImagesFromArray(arrys)
datanew.nrecords
datanew.dims.count
img = datanew.values().first()
image(img[:,:,0])
Explanation: We can now load this array directly as an Images object
End of explanation
datanew = tsc.loadImagesFromArray(arrys[:,:,:,0])
datanew.nrecords
datanew.dims.count
Explanation: When loading n-dimensional array data, the first dimension is assumed to index the images, so a 4D array will be treated as a collection of 3D volumes, and a 3D array will be treated as a collection of 2D images.
End of explanation
seriespath = pth.join(pth.dirname(pth.realpath(thunder.__file__)), 'utils/data/iris/')
data = tsc.loadSeries(seriespath + 'iris.txt', inputFormat='text', nkeys=1)
Explanation: Loading Series
A Series object is a distributed collection of one-dimensional arrays with tuple key identifiers. All arrays in a Series must have the same length. They can be loaded from flat text or binary files.
Text files must contain a line for each record, with numbers separated by spaces. The first numbers of each line will be interpreted as keys, and subsequent numbers will be interpreted as values. The number of keys is user-specified. As before, we'll load example series data from a small file included with Thunder purely for testing purposes (this is the iris dataset).
End of explanation
data.first()
Explanation: If data were split across multiple files, we could have also provided the folder name, rather than the file, and all file(s) of the given format would be loaded. Let's look at the first entry of the Series:
End of explanation
data.index
Explanation: The index is automatically calculated based on the length of the values array:
End of explanation
dataraw = sc.textFile(seriespath + 'iris.txt')
dataraw.first()
Explanation: For comparison, you can look at the first raw line of the text file:
End of explanation
data = tsc.loadSeries(seriespath + 'iris.bin', inputFormat='binary')
data.first()
Explanation: Flat binary files must store each record as a contiguous sequence of bytes, with a fixed size in bytes for each record, including both keys and values. The number and numerical type of keys and records are most convieniently specified in a configuration file in the same directory as the data, but can also be specified as input arguments. Here, we show two ways of loading a binary version of the same iris data loaded previously.
End of explanation
path = seriespath + 'iris.bin'
data = tsc.loadSeries(path, inputFormat='binary', nkeys=1, nvalues=4, keyType='float', valueType='float')
data.first()
Explanation: In this case, the number of keys and number of values in each record, along with the data types of the keys and values, are automatically read out from a conf.json file located in the same directory as iris.bin. This file has the following simple JSON format.
{
  "keytype": "int16", 
  "valuetype": "uint8", 
  "nkeys": 3, 
  "nvalues": 240
}
When a conf.json file is unavailable, these parameters can also be passed as arguments to the loadSeries method:
End of explanation
data = tsc.loadSeries(seriespath + '/iris.mat', inputFormat='mat', varName='data', minPartitions=5)
data.first()
data = tsc.loadSeries(seriespath + '/iris.npy', inputFormat='npy', minPartitions=5)
data.first()
Explanation: Flat binary files are a particularly convienient format when exporting large data sets from other scientific computing environments, e.g. Matlab. For example, the data loaded above was written out from within Matlab using
f = fopen('iris.bin','w')
fwrite(f, [[0:149]' data]','double')
Where data is a matrix containing the data, and we append a column for the indices. Note that the data must be transposed due to ordering conventions.
Both text and binary data can be loaded from a single file or multiple files, stored on a local file system, a networked file system, Amazon S3, or HDFS. To load multiple files at once, specify a directory as the filename, or a wildcard pattern.
You can load Series data from local arrays saved in either numpy npy or Matlab MAT format. This is particular useful for local use, or for distributing a smaller data set for performing intensive computations. In the latter case, the number of partitions should be set to approximately 2-3 times the number of cores avaialble on your cluster, so that different cores can work on different portions of the data.
End of explanation
arry = data.collectValuesAsArray()
arry.shape
datanew = tsc.loadSeriesFromArray(arry)
datanew.first()
datanew.nrecords
Explanation: Finally, like Images, a Series object can be constructed directly from a numpy array in memory. To see this, we'll collect the data we just loaded as an array, and then use it to create a new Series object
End of explanation
<END_TASK>