<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 6 - Data Compression
This short Notebook will introduce you to how to efficiently compress your data within SampleData datasets.        
<div class="alert alert-info">
**Note** 
Throughout this notebook, it will be assumed that the reader is familiar with the overview of the SampleData file format and data model presented in the [first notebook of this User Guide](./SampleData_Introduction.ipynb) of this User Guide.
</div>
Data compression with HDF5 and Pytables
HDF5 and data compression
HDF5 allows compression filters to be applied to datasets in a file to minimize the amount of space it consumes. These compression features allow to drastically improve the storage space required for you datasets, as well as the speed of I/O access to datasets, and can differ from one data item to another within the same HDF5 file. A detailed presentation of HDF5 compression possibilities is provided here. 
The two main ingredients that control compression performances for HDF5 datasets are the compression filters used (which compression algorithm, with which parameters), and the using a chunked layout for the data. This two features are briefly developed hereafter.
Pytables and data compression
The application of compression filters to HDF5 files with the SampleData class is handled by the Pytable package, on which is built the SampleData HDF5 interface. Pytables implements a specific containers class, the Filters class, to gather the various settings of the compression filters to apply to the datasets in a HDF5 file.
When using the SampleData class, you will have to specify this compression filter settings to class methods dedicated to data compression.  These settings are the parameters of the Pytables Filters class. These settings and their possible values are detailed in the next subsection. 
Available filter settings
The description given below of compression options available with SampleData/Pytables is exctracted from the Pytables documentation of the Filter class.
complevel (int) – Specifies a compression level for data. The allowed range is 0-9. A value of 0 (the default) disables compression.
complib (str) – Specifies the compression library to be used. Right now, zlib (the default), lzo, bzip2 and blosc are supported. Additional compressors for Blosc like blosc
Step1:  This file is zipped in the package to reduce its size. We will have to unzip it to use it and learn how to reduce its size with the SampleData methods. If you are just reading the documentation and not executing it, you may just skip this cell and the next one.
Step2:  Dataset presentation
In this tutorial, we will work on a copy of this dataset, to leave the original data unaltered. 
We will start by creating an autodeleting copy of the file, and print its content to discover its content.
Step3:  As you can see, this dataset already contains a rich content. It is a digital twin of a real polycristalline microstructure of a grade 2 Titanium sample, gathering both experimental and numerical data obtained through Diffraction Contrast Tomography imaging, and FFT-based mechanical simulation. 
This dataset has actually been constructed using the Microstructure class of the pymicro package, which is based on the SampleData class. The link between these classes will be discussed in the next tutorial.
This dataset contains only uncompressed data. We will try to reduce its size by using various compression methods on the large data items that it contains. You can see that most of them are stored in the 3DImage Group CellData. 
Apply compression settings for a specific array
We will start by compressing the grain_map Field data array of the CellData image. Let us look more closely on this data item
Step4:  We can see above that this data item is not compressed (complevel=0), and has a disk size of almost 2 Mb.
To apply a set of compression settings to this data item, you need to
Step5:  use the SampleData set_chunkshape_and_compression method with the dictionary and the name of the data item as arguments
Step6:  As you can see, the storage size of the data item has been greatly reduced, by more than 10 times (126 Kb vs 1.945 Mb), using this compression settings. Let us see what will change if we use different settings 
Step7:  As you may observe, is significantly affected by the choice of the compression level. The higher the compression level, the higher the compression ratio, but also the lower the I/O speed. On the other hand, you can also remark that, in the present case, using the shuffle filter deteriorates the compression ratio.
Let us try to with another data item
Step8:  On the opposite, for this second array, the shuffle filter improves significantly the compression ratio. However, in this case, you can see that the compression ratio achieved is much lower than for the grain_map array. 
<div class="alert alert-warning">
**Warning 1** 
The efficiency of compression algorithms in terms of compression ratio is strongly affected by the data itself (variety, value and position of the stored values in the array). Compression filters will not have the same behavior with all data arrays, as you have observed just above. Be aware of this fact, and do not hesitate to conduct tests to find the best settings for you datasets !
</div>
<div class="alert alert-warning">
**Warning 2** 
Whenever you change the compression or chunkshape settings of your datasets, the data item is re-created into the *SampleData* dataset, which may be costly in computational time. Be careful if you are dealing with very large data arrays and want to try out several settings to find the best I/O speed / compression ratio compromise, with the `set_chunkshape_and_compression` method. You may want to try on a subset of your large array to speed up the process.
</div>
Apply same compression settings for a serie of nodes
If you need to apply the same compression settings to a list of data items, you may use the set_nodes_compression_chunkshape. This method works exactly like set_chunkshape_and_compression, but take a list of nodenames as arguments instead of just one. The inputted compression settings are then applied to all the nodes in the list
Step9:  Lossy compression and data normalization
The compression filters used above preserve exactly the original values of the stored data. However, it is also possible with specific filters a lossy compression, which remove non relevant part of the data. As a result, data compression ratio is usually strongly increased, at the cost that stored data is no longer exactly equal to the inputed data array.
One of the most important feature of data array that increase their compressibility, is the presence of patterns in the data. If a value or a serie of values is repeated multiple times throughout the data array, data compression can be very efficient (the pattern can be stored only once). 
Numerical simulation and measurement tools usually output data in a standard simple or double precision floating point numbers, yiedling data arrays with values that have a lot of digits. Typically, these values are all different, and hence these arrays cannot be efficiently compressed.
The Amitex_stress_1 and Amitex_strain_1 data array are two tensor fields outputed by a continuum mechanics FFT-based solver, and typical fall into this category
Step10:  Lossy compression
Usually, the relevant precision of data is only of a few digits, so that many values of the array should be considered equal. The idea of lossy compression is to truncate values up to a desired precision, which  increases the number of equal values in a dataset and hence increases its compressibility. 
Lossy compression can be applied to floating point data arrays in SampleData datasets using the least_significant_digit compression setting. If you set the value of this option to $N$, the data will be truncated after the $N^{th}$ siginificant digit after the decimal point. Let us see an example.
Step11:  As you may observe, the compression ratio has been improved, and the retrieved values after lossy compression are effectively equal to the original array up to the third digit after the decimal point. 
We will now try to increase the compression ratio by reducing the number of conserved digits to 2
Step12:  As you can see, the compression ratio has again been improved, now close to 75%. Know, you know how to do to choose the best compromise between lost precision and compression ratio. 
Normalization to improve compression ratio
If you look more closely to the Amitex_stress_1 array values, you can observe that the value of this array have been outputed within a certain scale of values, which in particular impact the number of significant digits that come before the decimal point. Sometimes precision of the data would require less significant digits than its scale of representation.
In that case, storing the complete data array at its original scale is not necessary, and very inefficient in terms of data size. To optimize storage of such datasets, one can normalize them to a form with very few digits before the decimal point (1 or 2), and stored separately their scale to be able to revert the normalization operation when retrieiving data.
This allows to reduce the total number of significant digits of the data, and hence further improve the achievable compression ratio with lossy compression. 
The SampleData class allows you to aplly automatically this operation when applying compression settings to your dataset. All you have to do is add to the compression_option dictionary the key normalization with one of its possible values. 
To try it, we will close (and delete) our test dataset and recopy the original file, to apply normalization and lossy compression on the original raw data
Step13:  Standard Normalization
The standard normalization setting will center and reduce the data of an array $X$ by storing a new array $Y$ that is
Step14:  As you can see, the compression ratio has been strongly improved by this normalization operation, reaching 90%. 
When looking at the retrieved value after compression, you can see that depending on the field component that is observed, the relative precision loss varies. The third large component value error is less than 1%, which is consistent with the truncation to 2 significant digits. However, it is not the other components, that have smaller values by two or three orders of magnitude, and that are retrieved with larger errors.
This is explained by the fact that the standard normalization option scales the array as a whole. As a result, if there are large differencies in the scale of different components of a vector or tensor field, the precision of the smaller components will be less preserved.
Standard Normalization per components for vector/tensor fields
Another normalization option is available for SampleData field arrays, that allows to apply standard normalization individually to each component of a field in order to keep a constant relative precision for each component when applying lossy compression to the field data array.
To use this option, you will need to set the normalization value to standard_per_component
Step15:  As you can see, the error in the retrieved array is now less than 1% for each component of the field value. However, the cost was a reduced improvement of the compression ratio. 
Visualization of normalized data
Step16:  As you can see, the Amitex_stress_1 Attribute node data in the dataset XDMF file is now provided by a Function item type, involving three data array with the original field shape. This function computes
Step17:  Changing the chunksize of a node
Compressing all fields when adding Image or Mesh Groups
Changing the chunksize of a data array with SampleData is very simple. You just have to pass as a tuple the news shape of the chunks you want for your data array, and pass it as an argument to the set_chunkshape_and_compression or set_nodes_compression_chunkshape
Step18:  As you can see, the chunkshape has been changed, which has also affected the memory size of the compressed data array. We have indeed reduced the number of chunks in the dataset, which reduces the number of data to store. This modification can also improve or deteriorate the I/O speed of access to your data array in the dataset. The reader is once again refered to dedicated documents to know more ion this matter
Step19:  The node has been created with the desired chunkshape and compression filters.
Repacking files
We now recreate a new copy of the original dataset, and try to reduce the size of oll heavy data item, to reduce as much as possible the size of our dataset.
Step20:  Now that we have compressed a few of the items of our dataset, the disk size of its HDF5 file should have diminished. Let us check again the size of its data items, and of the file
Step21:  The file size has not changed, surprisingly, even if the large Amitex_stress_1 array has been shrinked from almost 50 Mo to roughly 5 Mo. This is due to a specific feature of HDF5 files
Step22:  You see that repacking the file has allowed to free some memory space and reduced its size. 
<div class="alert alert-info">
**Note** 
Note that the size of the file is larger than the size of data items printed by `print_dataset_content`. This extra size is the memory size occupied by the data array storing *Element Tags* for the mesh `grains_mesh`. Element tags are not printed by the printing methods as they can be very numerous and pollute the lecture of the printed information.
</div>
Once again, you should repack your file at carefully chosen times, as is it a very costly operation for large datasets. The SampleData class constructor has an autorepack option. If it is set to True, the file is automatically repacked when closing the dataset.
We can now close our dataset, and remove the original unarchived file
<ASSISTANT_TASK:>
Python Code:
from config import PYMICRO_EXAMPLES_DATA_DIR # import file directory path
import os
dataset_file = os.path.join(PYMICRO_EXAMPLES_DATA_DIR, 'example_microstructure') # test dataset file path
tar_file = os.path.join(PYMICRO_EXAMPLES_DATA_DIR, 'example_microstructure.tar.gz') # dataset archive path
Explanation: 6 - Data Compression
This short Notebook will introduce you to how to efficiently compress your data within SampleData datasets.        
<div class="alert alert-info">
**Note** 
Throughout this notebook, it will be assumed that the reader is familiar with the overview of the SampleData file format and data model presented in the [first notebook of this User Guide](./SampleData_Introduction.ipynb) of this User Guide.
</div>
Data compression with HDF5 and Pytables
HDF5 and data compression
HDF5 allows compression filters to be applied to datasets in a file to minimize the amount of space it consumes. These compression features allow to drastically improve the storage space required for you datasets, as well as the speed of I/O access to datasets, and can differ from one data item to another within the same HDF5 file. A detailed presentation of HDF5 compression possibilities is provided here. 
The two main ingredients that control compression performances for HDF5 datasets are the compression filters used (which compression algorithm, with which parameters), and the using a chunked layout for the data. This two features are briefly developed hereafter.
Pytables and data compression
The application of compression filters to HDF5 files with the SampleData class is handled by the Pytable package, on which is built the SampleData HDF5 interface. Pytables implements a specific containers class, the Filters class, to gather the various settings of the compression filters to apply to the datasets in a HDF5 file.
When using the SampleData class, you will have to specify this compression filter settings to class methods dedicated to data compression.  These settings are the parameters of the Pytables Filters class. These settings and their possible values are detailed in the next subsection. 
Available filter settings
The description given below of compression options available with SampleData/Pytables is exctracted from the Pytables documentation of the Filter class.
complevel (int) – Specifies a compression level for data. The allowed range is 0-9. A value of 0 (the default) disables compression.
complib (str) – Specifies the compression library to be used. Right now, zlib (the default), lzo, bzip2 and blosc are supported. Additional compressors for Blosc like blosc:blosclz (‘blosclz’ is the default in case the additional compressor is not specified), blosc:lz4, blosc:lz4hc, blosc:snappy, blosc:zlib and blosc:zstd are supported too. Specifying a compression library which is not available in the system issues a FiltersWarning and sets the library to the default one.
shuffle (bool) – Whether or not to use the Shuffle filter in the HDF5 library. This is normally used to improve the compression ratio. A false value disables shuffling and a true one enables it. The default value depends on whether compression is enabled or not; if compression is enabled, shuffling defaults to be enabled, else shuffling is disabled. Shuffling can only be used when compression is enabled.
bitshuffle (bool) – Whether or not to use the BitShuffle filter in the Blosc library. This is normally used to improve the compression ratio. A false value disables bitshuffling and a true one enables it. The default value is disabled.
fletcher32 (bool) – Whether or not to use the Fletcher32 filter in the HDF5 library. This is used to add a checksum on each data chunk. A false value (the default) disables the checksum.
least_significant_digit (int) – If specified, data will be truncated (quantized). In conjunction with enabling compression, this produces ‘lossy’, but significantly more efficient compression. For example, if least_significant_digit=1, data will be quantized using around(scale*data)/scale, where scale = 2^bits, and bits is determined so that a precision of 0.1 is retained (in this case bits=4). Default is None, or no quantization.
Chunked storage layout
Compressed data is stored in a data array of an HDF5 dataset using a chunked storage mechanism. When chunked storage is used, the data array is split into equally sized chunks each of which is stored separately in the file, as illustred on the diagram below. Compression is applied to each individual chunk. When an I/O operation is performed on a subset of the data array, only chunks that include data from the subset participate in I/O and need to be uncompressed or compressed.
Chunking data allows to:
Generally improve, sometimes drastically, the I/O performance of datasets. This comes from the fact that the chunked layout removes the reading speed anisotropy for data array that depends along which dimension its elements are read (i.e the same number of disk access are required when reading data in rows or columns).
Chunked storage also enables adding more data to a dataset without rewriting the whole dataset.
<img src="./Images/Tutorial_5/chuncked_layout.png" width="50%">
By default, data arrays are stored with a chunked layout in SampleData datasets. The size of chunks is the key parameter that controls the impact on I/O performances for chunked datasets. The shape of chunks is computed automatically by the Pytable package, providing a value yielding generally good I/O performances. if you need to go further in the I/O optimization, you may consult the Pytables documentation page dedicated to compression optimization for I/O speed and storage space. In addition, it is highly recommended to read this document in order to be able to efficiently optimize I/O and storage performances for your chunked datasets. These performance issues will not be discussed in this tutorial.
Compressing your datasets with SampleData
Within Sampledata, data compression can be applied to:
data arrays
structured data arrays
field data arrays
There are two ways to control the compression settings of your SampleData data arrays:
Providing compression settings to data item creation methods
Using the set_chunkshape_and_compression and set_nodes_compression_chunkshape methods
The compression options dictionary
In both cases, you will have to pass the various settings of the compression filter you want to apply to your data to the appropriate SampleData method. All of these methods accept for that purpose a compression_options argument, which must be a dictionary. Its keys and associated values can be chosen among the ones listed in the Available filter settings subsection above.
Compress already existing data arrays
We will start by looking at how we can change compression settings of already existing data in SampleData datasets.
For that, we will use a material science dataset that is part of the Pymicro example datasets.
End of explanation
# Save current directory
cwd = os.getcwd()
# move to example data directory
os.chdir(PYMICRO_EXAMPLES_DATA_DIR)
# unarchive the dataset
os.system(f'tar -xvf {tar_file}')
# get back to UserGuide directory
os.chdir(cwd)
Explanation: This file is zipped in the package to reduce its size. We will have to unzip it to use it and learn how to reduce its size with the SampleData methods. If you are just reading the documentation and not executing it, you may just skip this cell and the next one.
End of explanation
# import SampleData class
from pymicro.core.samples import SampleData as SD
# import Numpy
import numpy as np
# Create a copy of the existing dataset
data = SD.copy_sample(src_sample_file=dataset_file, dst_sample_file='Test_compression', autodelete=True,
                      get_object=True, overwrite=True)
print(data)
data.get_file_disk_size()
Explanation: Dataset presentation
In this tutorial, we will work on a copy of this dataset, to leave the original data unaltered. 
We will start by creating an autodeleting copy of the file, and print its content to discover its content.
End of explanation
data.print_node_info('grain_map')
Explanation: As you can see, this dataset already contains a rich content. It is a digital twin of a real polycristalline microstructure of a grade 2 Titanium sample, gathering both experimental and numerical data obtained through Diffraction Contrast Tomography imaging, and FFT-based mechanical simulation. 
This dataset has actually been constructed using the Microstructure class of the pymicro package, which is based on the SampleData class. The link between these classes will be discussed in the next tutorial.
This dataset contains only uncompressed data. We will try to reduce its size by using various compression methods on the large data items that it contains. You can see that most of them are stored in the 3DImage Group CellData. 
Apply compression settings for a specific array
We will start by compressing the grain_map Field data array of the CellData image. Let us look more closely on this data item:
End of explanation
compression_options = {'complib':'zlib', 'complevel':1}
Explanation: We can see above that this data item is not compressed (complevel=0), and has a disk size of almost 2 Mb.
To apply a set of compression settings to this data item, you need to:
create a dictionary specifying the compression settings:
End of explanation
data.set_chunkshape_and_compression(nodename='grain_map', compression_options=compression_options)
data.get_node_disk_size('grain_map')
data.print_node_compression_info('grain_map')
Explanation: use the SampleData set_chunkshape_and_compression method with the dictionary and the name of the data item as arguments
End of explanation
# No `shuffle` option:
print('\nUsing the shuffle option, with the zlib compressor and a compression level of 1:')
compression_options = {'complib':'zlib', 'complevel':1, 'shuffle':True}
data.set_chunkshape_and_compression(nodename='grain_map', compression_options=compression_options)
data.get_node_disk_size('grain_map')
# No `shuffle` option:
print('\nUsing no shuffle option, with the zlib compressor and a compression level of 9:')
compression_options = {'complib':'zlib', 'complevel':9, 'shuffle':False}
data.set_chunkshape_and_compression(nodename='grain_map', compression_options=compression_options)
data.get_node_disk_size('grain_map')
# No `shuffle` option:
print('\nUsing the shuffle option, with the lzo compressor and a compression level of 1:')
compression_options = {'complib':'lzo', 'complevel':1, 'shuffle':True}
data.set_chunkshape_and_compression(nodename='grain_map', compression_options=compression_options)
data.get_node_disk_size('grain_map')
# No `shuffle` option:
print('\nUsing no shuffle option, with the lzo compressor and a compression level of 1:')
compression_options = {'complib':'lzo', 'complevel':1, 'shuffle':False}
data.set_chunkshape_and_compression(nodename='grain_map', compression_options=compression_options)
data.get_node_disk_size('grain_map')
Explanation: As you can see, the storage size of the data item has been greatly reduced, by more than 10 times (126 Kb vs 1.945 Mb), using this compression settings. Let us see what will change if we use different settings :
End of explanation
data.print_node_info('Amitex_stress_1')
# No `shuffle` option:
print('\nUsing the shuffle option, with the zlib compressor and a compression level of 1:')
compression_options = {'complib':'zlib', 'complevel':1, 'shuffle':True}
data.set_chunkshape_and_compression(nodename='Amitex_stress_1', compression_options=compression_options)
data.get_node_disk_size('Amitex_stress_1')
# No `shuffle` option:
print('\nUsing no shuffle option, with the zlib compressor and a compression level of 1:')
compression_options = {'complib':'zlib', 'complevel':1, 'shuffle':False}
data.set_chunkshape_and_compression(nodename='Amitex_stress_1', compression_options=compression_options)
data.get_node_disk_size('Amitex_stress_1')
Explanation: As you may observe, is significantly affected by the choice of the compression level. The higher the compression level, the higher the compression ratio, but also the lower the I/O speed. On the other hand, you can also remark that, in the present case, using the shuffle filter deteriorates the compression ratio.
Let us try to with another data item:
End of explanation
# Print current size of disks and their compression settings
data.get_node_disk_size('grain_map_raw')
data.print_node_compression_info('grain_map_raw')
data.get_node_disk_size('uncertainty_map')
data.print_node_compression_info('uncertainty_map')
data.get_node_disk_size('mask')
data.print_node_compression_info('mask')
# Compress datasets
compression_options = {'complib':'zlib', 'complevel':9, 'shuffle':True}
data.set_nodes_compression_chunkshape(node_list=['grain_map_raw', 'uncertainty_map', 'mask'], 
                                      compression_options=compression_options)
# Print new size of disks and their compression settings
data.get_node_disk_size('grain_map_raw')
data.print_node_compression_info('grain_map_raw')
data.get_node_disk_size('uncertainty_map')
data.print_node_compression_info('uncertainty_map')
data.get_node_disk_size('mask')
data.print_node_compression_info('mask')
Explanation: On the opposite, for this second array, the shuffle filter improves significantly the compression ratio. However, in this case, you can see that the compression ratio achieved is much lower than for the grain_map array. 
<div class="alert alert-warning">
**Warning 1** 
The efficiency of compression algorithms in terms of compression ratio is strongly affected by the data itself (variety, value and position of the stored values in the array). Compression filters will not have the same behavior with all data arrays, as you have observed just above. Be aware of this fact, and do not hesitate to conduct tests to find the best settings for you datasets !
</div>
<div class="alert alert-warning">
**Warning 2** 
Whenever you change the compression or chunkshape settings of your datasets, the data item is re-created into the *SampleData* dataset, which may be costly in computational time. Be careful if you are dealing with very large data arrays and want to try out several settings to find the best I/O speed / compression ratio compromise, with the `set_chunkshape_and_compression` method. You may want to try on a subset of your large array to speed up the process.
</div>
Apply same compression settings for a serie of nodes
If you need to apply the same compression settings to a list of data items, you may use the set_nodes_compression_chunkshape. This method works exactly like set_chunkshape_and_compression, but take a list of nodenames as arguments instead of just one. The inputted compression settings are then applied to all the nodes in the list:
End of explanation
import numpy as np
print(f"Data array `grain_map` has {data['grain_map'].size} elements,"
      f"and {np.unique(data['grain_map']).size} different values.\n")
print(f"Data array `Amitex_stress_1` has {data['Amitex_stress_1'].size} elements,"
      f"and {np.unique(data['Amitex_stress_1']).size} different values.\n")
Explanation: Lossy compression and data normalization
The compression filters used above preserve exactly the original values of the stored data. However, it is also possible with specific filters a lossy compression, which remove non relevant part of the data. As a result, data compression ratio is usually strongly increased, at the cost that stored data is no longer exactly equal to the inputed data array.
One of the most important feature of data array that increase their compressibility, is the presence of patterns in the data. If a value or a serie of values is repeated multiple times throughout the data array, data compression can be very efficient (the pattern can be stored only once). 
Numerical simulation and measurement tools usually output data in a standard simple or double precision floating point numbers, yiedling data arrays with values that have a lot of digits. Typically, these values are all different, and hence these arrays cannot be efficiently compressed.
The Amitex_stress_1 and Amitex_strain_1 data array are two tensor fields outputed by a continuum mechanics FFT-based solver, and typical fall into this category: they have almost no equal value or clear data pattern. 
As you can see above, the best achieved compression ratio is 60% while for the dataset grain_map, the compression is way more efficient, with a best ratio that climbs up to 97% (62 Kb with zlib compressor and compression level of 9, versus an initial data array of 1.945 Mb). This is due to the nature of the grain_map data array, which is a tridimensional map of grains identification number in microstructure of the Titanium sample represented by the dataset. It is hence an array containing a few integer values that are repeated many times.
Let us analyze these two data arrays values to illustrate this difference:
End of explanation
# We will store a value of an array to verify how it evolves after compression
original_value = data['Amitex_stress_1'][20,20,20]
# Apply lossy compression
data.get_node_disk_size('Amitex_stress_1')
# Set up compression settings with lossy compression: truncate after third digit adter decimal point
compression_options = {'complib':'zlib', 'complevel':9, 'shuffle':True,  'least_significant_digit':3}
data.set_chunkshape_and_compression(nodename='Amitex_stress_1', compression_options=compression_options)
data.get_node_disk_size('Amitex_stress_1')
# Get same value after lossy compression
new_value = data['Amitex_stress_1'][20,20,20]
print(f'Original array value: {original_value} \n'
      f'Array value after lossy compression: {new_value}')
Explanation: Lossy compression
Usually, the relevant precision of data is only of a few digits, so that many values of the array should be considered equal. The idea of lossy compression is to truncate values up to a desired precision, which  increases the number of equal values in a dataset and hence increases its compressibility. 
Lossy compression can be applied to floating point data arrays in SampleData datasets using the least_significant_digit compression setting. If you set the value of this option to $N$, the data will be truncated after the $N^{th}$ siginificant digit after the decimal point. Let us see an example.
End of explanation
# Set up compression settings with lossy compression: truncate after third digit adter decimal point
compression_options = {'complib':'zlib', 'complevel':9, 'shuffle':True,  'least_significant_digit':2}
data.set_chunkshape_and_compression(nodename='Amitex_stress_1', compression_options=compression_options)
data.get_node_disk_size('Amitex_stress_1')
# Get same value after lossy compression
new_value = data['Amitex_stress_1'][20,20,20]
print(f'Original array value: {original_value} \n'
      f'Array value after lossy compression 2 digits: {new_value}')
Explanation: As you may observe, the compression ratio has been improved, and the retrieved values after lossy compression are effectively equal to the original array up to the third digit after the decimal point. 
We will now try to increase the compression ratio by reducing the number of conserved digits to 2:
End of explanation
# removing dataset to recreate a copy
del data
# creating a copy of the dataset to try out lossy compression methods
data = SD.copy_sample(src_sample_file=dataset_file, dst_sample_file='Test_compression', autodelete=True,
                      get_object=True, overwrite=True)
Explanation: As you can see, the compression ratio has again been improved, now close to 75%. Know, you know how to do to choose the best compromise between lost precision and compression ratio. 
Normalization to improve compression ratio
If you look more closely to the Amitex_stress_1 array values, you can observe that the value of this array have been outputed within a certain scale of values, which in particular impact the number of significant digits that come before the decimal point. Sometimes precision of the data would require less significant digits than its scale of representation.
In that case, storing the complete data array at its original scale is not necessary, and very inefficient in terms of data size. To optimize storage of such datasets, one can normalize them to a form with very few digits before the decimal point (1 or 2), and stored separately their scale to be able to revert the normalization operation when retrieiving data.
This allows to reduce the total number of significant digits of the data, and hence further improve the achievable compression ratio with lossy compression. 
The SampleData class allows you to aplly automatically this operation when applying compression settings to your dataset. All you have to do is add to the compression_option dictionary the key normalization with one of its possible values. 
To try it, we will close (and delete) our test dataset and recopy the original file, to apply normalization and lossy compression on the original raw data:
End of explanation
# Set up compression settings with lossy compression: truncate after third digit adter decimal point
compression_options = {'complib':'zlib', 'complevel':9, 'shuffle':True,  'least_significant_digit':2,
                       'normalization':'standard'}
data.set_chunkshape_and_compression(nodename='Amitex_stress_1', compression_options=compression_options)
data.get_node_disk_size('Amitex_stress_1')
# Get same value after lossy compression
new_value = data['Amitex_stress_1'][20,20,20,:]
# Get in memory value of the node
memory_value = data.get_node('Amitex_stress_1', as_numpy=False)[20,20,20,:]
print(f'Original array value: {original_value} \n'
      f'Array value after normalization and lossy compression 2 digits: {new_value}',
      f'Value in memory: {memory_value}')
Explanation: Standard Normalization
The standard normalization setting will center and reduce the data of an array $X$ by storing a new array $Y$ that is:
$Y = \frac{X - \bar{X}}{\sigma(X)}$
where $\bar{X}$ and $\sigma(X)$ are respectively the mean and the standard deviation of the data array $X$. 
This operation reduces the number of significant digits before the decimal point to 1 or 2 for the large majority of the data array values. After standard normalization, lossy compression will yield much higher compression ratios for data array that have a non normalized scale. 
The SampleData class ensures that when data array are retrieved, or visualized, the user gets or sees the original data, with the normalization reverted.  
Let us try to apply it to our stress field Amitex_stress_1.
End of explanation
del data
data = SD.copy_sample(src_sample_file=dataset_file, dst_sample_file='Test_compression', autodelete=True,
                      get_object=True, overwrite=True)
# Set up compression settings with lossy compression: truncate after third digit adter decimal point
compression_options = {'complib':'zlib', 'complevel':9, 'shuffle':True,  'least_significant_digit':2,
                       'normalization':'standard_per_component'}
data.set_chunkshape_and_compression(nodename='Amitex_stress_1', compression_options=compression_options)
data.get_node_disk_size('Amitex_stress_1')
# Get same value after lossy compression
new_value = data['Amitex_stress_1'][20,20,20,:]
# Get in memory value of the node
memory_value = data.get_node('Amitex_stress_1', as_numpy=False)[20,20,20,:]
print(f'Original array value: {original_value} \n'
      f'Array value after normalization per component and lossy compression 2 digits: {new_value}\n',
      f'Value in memory: {memory_value}')
Explanation: As you can see, the compression ratio has been strongly improved by this normalization operation, reaching 90%. 
When looking at the retrieved value after compression, you can see that depending on the field component that is observed, the relative precision loss varies. The third large component value error is less than 1%, which is consistent with the truncation to 2 significant digits. However, it is not the other components, that have smaller values by two or three orders of magnitude, and that are retrieved with larger errors.
This is explained by the fact that the standard normalization option scales the array as a whole. As a result, if there are large differencies in the scale of different components of a vector or tensor field, the precision of the smaller components will be less preserved.
Standard Normalization per components for vector/tensor fields
Another normalization option is available for SampleData field arrays, that allows to apply standard normalization individually to each component of a field in order to keep a constant relative precision for each component when applying lossy compression to the field data array.
To use this option, you will need to set the normalization value to standard_per_component:
End of explanation
data.print_xdmf()
Explanation: As you can see, the error in the retrieved array is now less than 1% for each component of the field value. However, the cost was a reduced improvement of the compression ratio. 
Visualization of normalized data
End of explanation
data.get_node_disk_size('Amitex_stress_1')
data.get_node_disk_size('Amitex_stress_1_norm_std')
data.get_node_disk_size('Amitex_stress_1_norm_mean')
Explanation: As you can see, the Amitex_stress_1 Attribute node data in the dataset XDMF file is now provided by a Function item type, involving three data array with the original field shape. This function computes:
$X' = Y*\sigma(X) + \bar{X}$
where $*$ and $+$ are element-wise product and addition operators for multidimensional arrays. This operation allows to revert the component-wise normalization of data. The Paraview software is able to interpret this syntax of the XDMF format and hence, when visualizing data, you will see the values with the original scaling.
This operation required the creation of two large arrays in the dataset, that the store the mean and standard deviation of each component of the field, repeted for each spatial dimensions of the field data array. It is mandatory to allow visualization of the data with the right scaling in Paraview. However, as these array contain a very low amount of data ($2*N_c$: two times de number of components of the field), they can be very easily compressed and hence do not significantly affect the storage size of the data item, as you may see below:
End of explanation
data.print_node_compression_info('Amitex_stress_1')
data.get_node_disk_size('Amitex_stress_1')
# Change chunkshape of the array
compression_options = {'complib':'zlib', 'complevel':9, 'shuffle':True,  'least_significant_digit':2,
                       'normalization':'standard_per_component'}
data.set_chunkshape_and_compression(nodename='Amitex_stress_1', chunkshape=(10,10,10,6),
                                   compression_options=compression_options)
data.get_node_disk_size('Amitex_stress_1')
data.print_node_compression_info('Amitex_stress_1')
Explanation: Changing the chunksize of a node
Compressing all fields when adding Image or Mesh Groups
Changing the chunksize of a data array with SampleData is very simple. You just have to pass as a tuple the news shape of the chunks you want for your data array, and pass it as an argument to the set_chunkshape_and_compression or set_nodes_compression_chunkshape:
End of explanation
# removing dataset to recreate a copy
del data
# creating a copy of the dataset to try out lossy compression methods
data = SD.copy_sample(src_sample_file=dataset_file, dst_sample_file='Test_compression', autodelete=True,
                      get_object=True, overwrite=True)
# getting the `orientation_map` array
array = data['Amitex_stress_1']
# create a new field for the CellData image group with the `orientation_map` array and add compression settings
compression_options = {'complib':'zlib', 'complevel':1, 'shuffle':True,  'least_significant_digit':2,
                       'normalization':'standard'}
new_cshape = (10,10,10,3)
# Add data array as field of the CellData Image Group
data.add_field(gridname='CellData', fieldname='test_compression', indexname='testC', array=array,
              chunkshape=new_cshape, compression_options=compression_options, replace=True)
# Check size and settings of new field
data.print_node_info('testC')
data.get_node_disk_size('testC')
data.print_node_compression_info('testC')
Explanation: As you can see, the chunkshape has been changed, which has also affected the memory size of the compressed data array. We have indeed reduced the number of chunks in the dataset, which reduces the number of data to store. This modification can also improve or deteriorate the I/O speed of access to your data array in the dataset. The reader is once again refered to dedicated documents to know more ion this matter: here and here.
Compression data and setting chunkshape upon creation of data items
Until here we have only modified the compression settings of already existing data items. In this process, the data items are replaced by the new compressed version of the data, which is a costly operation. For this reason, if they are known in advance, it is best to apply the compression filters and appropriate chunkshape when creating the data item.
If you have read through all the tutorials of this user guide, you should know all the method that allow to create data items in your datasets, like add_data_array, add_field, ædd_mesh... All of these methods accept the two arguments chunkshape and compression_options, that work exaclty as for the set_chunkshape_and_compression or set_nodes_compression_chunkshape methods. You hence use them to create your data items directly with the appropriate compression settings.
Let us see an example. We will get an array from our dataset, and try to recreate it with a new name and some data compression:
End of explanation
# removing dataset to recreate a copy
del data
# creating a copy of the dataset to try out lossy compression methods
data = SD.copy_sample(src_sample_file=dataset_file, dst_sample_file='Test_compression', autodelete=True,
                      get_object=True, overwrite=True)
compression_options1 = {'complib':'zlib', 'complevel':9, 'shuffle':True,  'least_significant_digit':2,
                       'normalization':'standard'}
compression_options2 = {'complib':'zlib', 'complevel':9, 'shuffle':True}
data.set_chunkshape_and_compression(nodename='Amitex_stress_1', compression_options=compression_options1)
data.set_nodes_compression_chunkshape(node_list=['grain_map', 'grain_map_raw','mask'],
                                      compression_options=compression_options2)
Explanation: The node has been created with the desired chunkshape and compression filters.
Repacking files
We now recreate a new copy of the original dataset, and try to reduce the size of oll heavy data item, to reduce as much as possible the size of our dataset.
End of explanation
data.print_dataset_content(short=True)
data.get_file_disk_size()
Explanation: Now that we have compressed a few of the items of our dataset, the disk size of its HDF5 file should have diminished. Let us check again the size of its data items, and of the file:
End of explanation
data.repack_h5file()
data.get_file_disk_size()
Explanation: The file size has not changed, surprisingly, even if the large Amitex_stress_1 array has been shrinked from almost 50 Mo to roughly 5 Mo. This is due to a specific feature of HDF5 files: they do not free up the memory space that they have used in the past. The memory space remains associated to the file, and is used in priority when new data is written into the dataset. 
After changing the compression settings of one or several nodes in your dataset, if that induced a reduction of your actual data memory size, and that you want your file to be smaller on disk. To retrieve the fried up memory spacae, you may repack your file (overwrite it with a copy of itself, that has just the size require to store all actual data).
To do that, you may use the SampleData method repack_h5file:
End of explanation
# remove SampleData instance
del data
os.remove(dataset_file+'.h5')
os.remove(dataset_file+'.xdmf')
Explanation: You see that repacking the file has allowed to free some memory space and reduced its size. 
<div class="alert alert-info">
**Note** 
Note that the size of the file is larger than the size of data items printed by `print_dataset_content`. This extra size is the memory size occupied by the data array storing *Element Tags* for the mesh `grains_mesh`. Element tags are not printed by the printing methods as they can be very numerous and pollute the lecture of the printed information.
</div>
Once again, you should repack your file at carefully chosen times, as is it a very costly operation for large datasets. The SampleData class constructor has an autorepack option. If it is set to True, the file is automatically repacked when closing the dataset.
We can now close our dataset, and remove the original unarchived file:
End of explanation
<END_TASK>