<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 Introduction to ASDF
Paper - ASDF
Step1:  Storing Arrays
Step2:  Schema validation
ASDF prevents files from being instantiated or saved with data not supported by a given schema.
Step3:  Data Sharing
Overlap between data sets can be stored without duplication.
Step4:  Streaming Data
Step5:  Explosive Storage
Data can be optionally stored in the header (if it's reasonably small) or exploded into separate files (if large binary).
Step6:  Data Provenance
Step7:  Compression
Step8:  Custom types
Example
Step9:  verify that time matches astropy type
Step10:  Units
ASDF is capable of storing scientific units via astropy schemas.
Step11:  verify variables load - comes as dictionary
<ASSISTANT_TASK:>
Python Code:
mkdir -p hello_world
from asdf import AsdfFile
# Make the tree structure, and create a AsdfFile from it.
tree = {'hello': 'world'}
ff = AsdfFile(tree)
ff.write_to("hello_world/test.asdf")
# You can also make the AsdfFile first, and modify its tree directly:
ff = AsdfFile()
ff.tree['hello'] = 'world'
ff.write_to("hello_world/test_hello_world.asdf")
cat hello_world/test.asdf
cat hello_world/test_hello_world.asdf
Explanation: Introduction to ASDF
Paper - ASDF: A new data format for astronomy
http://dx.doi.org/10.1016/j.ascom.2015.06.004
Why another data format?
Shortcomings of FITS are becoming burdensome: 
* restrictions on keyword name length led to restricting the order of polynomials used (not enough characters for more than 1 digit)
* Namespace collisions led to lack of transformation concatenation
* lack of simple grouping structure
* Conventions for workarounds have been implimented, but these are not standardized
What about alternatives?
VOTable
Strengths: XML, can leverage correspoonding reader and validation libraies.
Weaknesses
No intrinsic support for efficiently handling binary data
can't do memory mapping for files too large to fit in memory
can refer to FITS, but doesn't solve FITS's problems
Does not allow for additional structured metadata beyond description of tabular data (free text wouldn't be sufficient to describe World Coordinate System transformations)
HDF5
Strengths
Flexible, used by a much broader community, perhaps strongest alternative to FITS
Weaknesses (relative to FITS)
Entirely binary - FITS headers were are atleast human readable.
All inspection must be done through HDF5 software
FITS is largely self documenting, while HDF5 spec is lengthy and complex (125 pages)
Complexity results in ~1 implimenation.. more likely to deviate from spec
HDF format kept changing, so it was less appropriate for archival format
Does not support simpler, text based data files
HDF5 Abstract Data Model was not flexible enough to represent generalized WCS
Arbitrary restriction on nesting of data structures
Data types do not include a variable-length mapping datatype (analogous to dict or js object)
Groups can not be nested in variable-length arrays, only within each other
Compound data dtypes cannoot contain other "compound" types or variable-length arrays
Problems with others
CDF 
purely binary
no supportfor grouping, hierarchical structures
no references
netCDF 
Also purely binary
Does not support references or compression
V.3 does not support hiearchical grouping
V.4 is layered on hdf5, so inherits most of HDF5's issues, although fewer features and simpler api
Starlink - does not support 64-bit dimension sizes, table structures organized by row, nor compression
XDF -  like VOTable, suffers from inability to store raw binary, dev halted in 2006
FITSXML - embeds FITS in xml, but shares similar restrictions wrt grouping and metadata
ASDF Design goals
Main focus is on data interchange and archive sutability while maintinaing level of efficiency of FITS. They are not prioritizing throughput - expect tools to convert from ASDF to HDF5 where necessary.
Structure
Intrinsic hierarchical structure - syntax makes structure apparent
Human-readable
Based on an existing standard
Support for references - can refer to same object from multiple locations in the file
No arbitrary limits where possible
Efficient updating
Qualities for numerical Data
Support both text and binary, where binary may be stored raw. Text should be human-editable
Machine independent (endianess)
Structured - binary fits insised of structured hierachy
Multiple binary sections possible
n-dimensional arrays and tables
Intrinsic support for reading/writing streams
Interoperability
Explicit versioning for both the format and the individual stritures within it
Explicit extensibiility - domain specific extensions without interference
Validation - support a schema language that enables semantic validation of type and structure of objects. leverage existing validation toolsets. Enable schema validation for domain-specific extensions
ASDF Structure
One-line header indicating asdf and version
YAML "tree": Yet Another Markup Language (YAML)
* natively supports strings, numbers, booleans, null
* arbitrary nesting of mappings, sequences
* Similar to json, but structure more concise and human readable - can use indentation and/or delimiters
* has explicit type designations (YAML tags)
    * example: !wcs/steps -> validation performed against schema definition for that tag
* Supported by major programming languages
* Supports references 
    * "anchors": object->label
    * "alias": label->object
Binary Sections: Defined by supportin yaml schemas
Motivation
Kameleon is known as a data standard for the space weather community. Operationally, that means we 
write conversion software that puts the scientist's model results into cdf or hdf5 format
provide tools for cracking open and interpolating from the resulting files
we host model results for the public (read scientists') consumption.
Problems with our current standard
At the time the CCMC started, we chose to provide a common interface into the model results and to add our own metadata to the hosted results. There are a few problems with this approach
this approach is very labor intensive - unless the Kameleon developers add the model, it doesn't get supported.
the demand for bringing in new models is growing and we are constantly being asked to support more models. 
The cdf/hdf5 format requires the user to have specialized software just to view metadata.
While cdf/hdf5 standard is highly optimized for heavy models, it is overkill for many others.
Meanwhile.. IDL ROR Vis
For years, Lutz has been developing IDL readers and interpolators for all models currently in the Runs-on-Request system. Lutz's scripts convert the model results into a general N-D data structure, which his interpolators understand...
 Idea: use Lutz's format, but store results in ASDF .
Defining a format?
Our goal is to make it easy to bring space weather models into kameleon. We do this by either:
defining a reader/interpolator in code
by explicitly defining a file format (asdf)
Since Lutz has already created a format for reading the models into IDL, we are targeting a file format for him to write to.
Why ASDF
https://asdf.readthedocs.io/en/latest/
A Flexible file format for scientific data
Uses YAML to store metadata and provenance info.
Arrays stored in the same file or separately
On Load, multiple all files treated as one
Hello World
End of explanation
mkdir -p array_storage
from asdf import AsdfFile
import numpy as np
tree = {'my_array': np.random.rand(8, 8)}
ff = AsdfFile(tree)
ff.write_to("array_storage/test_arrays.asdf")
cat array_storage/test_arrays.asdf
Explanation: Storing Arrays
End of explanation
from asdf import ValidationError
from asdf import AsdfFile
tree = {'data': 'Not an array'}
try:
    AsdfFile(tree)
except:
    raise ValidationError('data needs an array!')
Explanation: Schema validation
ASDF prevents files from being instantiated or saved with data not supported by a given schema.
End of explanation
mkdir -p data_sharing
from asdf import AsdfFile
import numpy as np
my_array = np.random.rand(8, 8)
subset = my_array[2:4,3:6]
tree = {
    'my_array': my_array,
    'subset':   subset
}
ff = AsdfFile(tree)
ff.write_to("data_sharing/test_overlap.asdf")
cat data_sharing/test_overlap.asdf
Explanation: Data Sharing
Overlap between data sets can be stored without duplication.
End of explanation
mkdir -p streaming_data
from asdf import AsdfFile, Stream
import numpy as np
tree = {
    # Each "row" of data will have 128 entries.
    'my_stream': Stream([128], np.float64)
}
ff = AsdfFile(tree)
with open('streaming_data/stream_test.asdf', 'wb') as fd:
    ff.write_to(fd)
    # Write 100 rows of data, one row at a time.  ``write``
    # expects the raw binary bytes, not an array, so we use
    # ``tostring()``.
    for i in range(10):
        fd.write(np.array([i] * 128, np.float64).tostring())
cat streaming_data/stream_test.asdf
Explanation: Streaming Data
End of explanation
mkdir -p exploded_data
from asdf import AsdfFile
import numpy as np
my_array = np.random.rand(3, 4)
tree = {'my_array': my_array}
my_big_array = np.random.rand(8, 8)
tree['my_big_array'] = my_big_array
ff = AsdfFile(tree)
ff.set_array_storage(my_array, 'inline')
ff.set_array_storage(my_big_array, 'external')
ff.write_to("exploded_data/test_exploded.asdf")
# Or for every block:
# ff.write_to("test.asdf", all_array_storage='external')
ls exploded_data/
cat exploded_data/test_exploded.asdf
cat exploded_data/test_exploded0000.asdf
Explanation: Explosive Storage
Data can be optionally stored in the header (if it's reasonably small) or exploded into separate files (if large binary).
End of explanation
mkdir -p provenance
from asdf import AsdfFile
import numpy as np
tree = {
    'some_random_data': np.random.rand(5, 5)
}
ff = AsdfFile(tree)
ff.add_history_entry(
    u"Initial random numbers",
    {u'name': u'asdf examples',
     u'author': u'John Q. Public',
     u'homepage': u'http://github.com/spacetelescope/asdf',
     u'version': u'0.1',
    u'spase_dict': {u'resource_id': 5}})
ff.write_to('provenance/provenance.asdf')
cat provenance/provenance.asdf
Explanation: Data Provenance
End of explanation
mkdir -p compression
from asdf import AsdfFile
import numpy as np
x = np.linspace(-20, 20, 30)
y = np.linspace(-30, 30, 50)
xx,yy = np.meshgrid(x,y)
tree = dict(variables = dict(x = xx,
                             y = yy
                            )
           )
ff = AsdfFile(tree)
ff.write_to("compression/uncompressed_data.asdf", all_array_compression=None)
ff.write_to("compression/compressed_data.asdf", all_array_compression='bzp2')
import os
print 'uncompressed:', os.path.getsize("compression/uncompressed_data.asdf"), 'bytes'
print 'compressed (bz2):', os.path.getsize("compression/compressed_data.asdf"), 'bytes'
Explanation: Compression
End of explanation
mkdir -p time
from asdf import AsdfFile
from astropy.time import Time 
astrot = Time('2016-10-3')
from asdf.tags.time import TimeType
tree = {'my_time': astrot}
ff = AsdfFile(tree)
ff.write_to("time/test_time.asdf")
ff.close()
cat time/test_time.asdf
Explanation: Custom types
Example: Astropy Time
End of explanation
sample_time =  AsdfFile.open('time/test_time.asdf')
my_time = sample_time.tree['my_time']
type(my_time) == type(astrot)
Explanation: verify that time matches astropy type
End of explanation
mkdir -p units
from astropy import units as u
rho_unit = u.kg*u.cm**-3
density = np.linspace(0, 11, 5)*rho_unit
density.unit
from asdf import AsdfFile
tree = dict(variables=dict(density = dict(data=density.value, unit = density.unit)))
           
ff = AsdfFile(tree)
ff.set_array_storage(density, 'inline')
ff.write_to("units/units_test.asdf", all_array_compression=None)
ff.close()
cat units/units_test.asdf
Explanation: Units
ASDF is capable of storing scientific units via astropy schemas.
End of explanation
units_file =  AsdfFile.open('units/units_test.asdf')
rho = units_file.tree['variables']['density']
rho
Explanation: verify variables load - comes as dictionary
End of explanation
<END_TASK>