<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 Histogrammar basic tutorial
Histogrammar is a Python package that allows you to make histograms from numpy arrays, and pandas and spark dataframes. (There is also a scala backend for Histogrammar.) 
This basic tutorial shows how to
Step1:  Data generation
Let's first load some data!
Step2:  Let's fill a histogram!
Histogrammar treats histograms as objects. You will see this has various advantages.
Let's fill a simple histogram with a numpy array.
Step3:  Histogrammar also supports open-ended histograms, which are sparsely represented. Open-ended histograms are used when you have a distribution of known scale (bin width) but unknown domain (lowest and highest bin index). Bins in a sparse histogram only get created and filled if the corresponding data points are encountered. 
A sparse histogram has a binWidth, and optionally an origin parameter. The origin is the left edge of the bin whose index is 0 and is set to 0.0 by default. Sparse histograms are nice if you don't want to restrict the range, for example for tracking data distributions over time, which may have large, sudden outliers.
Step4:  Filling from a dataframe
Let's make the same 1d (sparse) histogram directly from a (pandas) dataframe.
Step5:  When importing histogrammar, pandas (and spark) dataframes get extra functions to create histograms that all start with "hg_". For example
Step6:  Handy histogram methods
For any 1-dimensional histogram extract the bin entries, edges and centers as follows
Step7:  Irregular bin histogram variants
There are two other open-ended histogram variants in addition to the SparselyBin we have seen before. Whereas SparselyBin is used when bins have equal width, the others offer similar alternatives to a single fixed bin width.
There are two ways
Step8:  Note the slightly different plotting style for CentrallyBin histograms (e.g. x-axis labels are central values instead of edges).
Multi-dimensional histograms
Let's make a multi-dimensional histogram. In Histogrammar, a multi-dimensional histogram is composed as two recursive histograms. 
We will use histograms with irregular binning in this example.
Step9:  Accessing bin entries
For most 2+ dimensional histograms, one can get the bin entries and centers as follows
Step10:  Accessing a sub-histogram
Depending on the histogram type of the first axis, hg.Bin or other, one can access the sub-histograms directly from
Step11:  Histogram types recap
So far we have covered the histogram types
Step12:  Categorize histograms also accept booleans
Step13:  Other histogram functionality
There are several more histogram types
Step14:  Stack histograms are useful to make efficiency curves.
With all these histograms you can make multi-dimensional histograms. For example, you can evaluate the mean and standard deviation of one feature as a function of bins of another feature. (A "profile" plot, similar to a box plot.)
Step15:  Convenience functions
There are several convenience functions to make such composed histograms. These are
Step16:  Overview of histograms
Here you can find the list of all available histograms and aggregators and how to use each one
Step17:  Storage
Histograms can be easily stored and retrieved in/from the json format.
<ASSISTANT_TASK:>
Python Code:
%%capture
# install histogrammar (if not installed yet)
import sys
!"{sys.executable}" -m pip install histogrammar
import histogrammar as hg
import pandas as pd
import numpy as np
import matplotlib
Explanation: Histogrammar basic tutorial
Histogrammar is a Python package that allows you to make histograms from numpy arrays, and pandas and spark dataframes. (There is also a scala backend for Histogrammar.) 
This basic tutorial shows how to:
- make histograms with numpy arrays and pandas dataframes, 
- plot them, 
- make multi-dimensional histograms,
- the various histogram types,
- to make many histograms at ones,
- and store and retrieve them. 
Enjoy!
End of explanation
# open a pandas dataframe for use below
from histogrammar import resources
df = pd.read_csv(resources.data("test.csv.gz"), parse_dates=["date"])
df.head()
Explanation: Data generation
Let's first load some data!
End of explanation
# this creates a histogram with 100 even-sized bins in the (closed) range [-5, 5]
hist1 = hg.Bin(num=100, low=-5, high=5)
# filling it with one data point:
hist1.fill(0.5)
hist1.entries
# filling the histogram with an array:
hist1.fill.numpy(np.random.normal(size=10000))
hist1.entries
# let's plot it
hist1.plot.matplotlib();
# Alternatively, you can call this to make the same histogram:
# hist1 = hg.Histogram(num=100, low=-5, high=5)
Explanation: Let's fill a histogram!
Histogrammar treats histograms as objects. You will see this has various advantages.
Let's fill a simple histogram with a numpy array.
End of explanation
hist2 = hg.SparselyBin(binWidth=10, origin=0)
hist2.fill.numpy(df['age'].values)
hist2.plot.matplotlib();
# Alternatively, you can call this to make the same histogram:
# hist2 = hg.SparselyHistogram(binWidth=10)
Explanation: Histogrammar also supports open-ended histograms, which are sparsely represented. Open-ended histograms are used when you have a distribution of known scale (bin width) but unknown domain (lowest and highest bin index). Bins in a sparse histogram only get created and filled if the corresponding data points are encountered. 
A sparse histogram has a binWidth, and optionally an origin parameter. The origin is the left edge of the bin whose index is 0 and is set to 0.0 by default. Sparse histograms are nice if you don't want to restrict the range, for example for tracking data distributions over time, which may have large, sudden outliers.
End of explanation
hist3 = hg.SparselyBin(binWidth=10, origin=0, quantity='age')
hist3.fill.numpy(df)
hist3.plot.matplotlib();
Explanation: Filling from a dataframe
Let's make the same 1d (sparse) histogram directly from a (pandas) dataframe.
End of explanation
# Alternatively, do:
hist3 = df.hg_SparselyBin(binWidth=10, origin=0, quantity='age')
# ... where hist3 automatically picks up column age from the dataframe, 
# ... and does not need to be filled by calling fill.numpy() explicitly.
Explanation: When importing histogrammar, pandas (and spark) dataframes get extra functions to create histograms that all start with "hg_". For example: hg_Bin or hg_SparselyBin.
Note that the column "age" is picked by setting quantity="age", and also that the filling step is done automatically.
End of explanation
# full range of bin entries, and those in a specified range:
(hist3.bin_entries(), hist3.bin_entries(low=30, high=80))
# full range of bin edges, and those in a specified range:
(hist3.bin_edges(), hist3.bin_edges(low=31, high=71))
# full range of bin centers, and those in a specified range:
(hist3.bin_centers(), hist3.bin_centers(low=31, high=80))
hsum = hist2 + hist3
hsum.entries
hsum *= 4
hsum.entries
Explanation: Handy histogram methods
For any 1-dimensional histogram extract the bin entries, edges and centers as follows:
End of explanation
hist4 = hg.CentrallyBin(centers=[15, 25, 35, 45, 55, 65, 75, 85, 95], quantity='age')
hist4.fill.numpy(df)
hist4.plot.matplotlib();
hist4.bin_edges()
Explanation: Irregular bin histogram variants
There are two other open-ended histogram variants in addition to the SparselyBin we have seen before. Whereas SparselyBin is used when bins have equal width, the others offer similar alternatives to a single fixed bin width.
There are two ways:
- CentrallyBin histograms, defined by specifying bin centers;
- IrregularlyBin histograms, with irregular bin edges.
They both partition a space into irregular subdomains with no gaps and no overlaps.
End of explanation
edges1 = [-100, -75, -50, -25, 0, 25, 50, 75, 100]
edges2 = [-200, -150, -100, -50, 0, 50, 100, 150, 200]
hist1 = hg.IrregularlyBin(edges=edges1, quantity='latitude')
hist2 = hg.IrregularlyBin(edges=edges2, quantity='longitude', value=hist1)
# for 3 dimensions or higher simply add the 2-dim histogram to the value argument
hist3 = hg.SparselyBin(binWidth=10, quantity='age', value=hist2)
hist1.bin_centers()
hist2.bin_centers()
hist2.fill.numpy(df)
hist2.plot.matplotlib();
# number of dimensions per histogram
(hist1.n_dim, hist2.n_dim, hist3.n_dim)
Explanation: Note the slightly different plotting style for CentrallyBin histograms (e.g. x-axis labels are central values instead of edges).
Multi-dimensional histograms
Let's make a multi-dimensional histogram. In Histogrammar, a multi-dimensional histogram is composed as two recursive histograms. 
We will use histograms with irregular binning in this example.
End of explanation
from histogrammar.plot.hist_numpy import get_2dgrid
x_labels, y_labels, grid = get_2dgrid(hist2)
y_labels, grid
Explanation: Accessing bin entries
For most 2+ dimensional histograms, one can get the bin entries and centers as follows:
End of explanation
# Acces sub-histograms from IrregularlyBin from hist.bins
# The first item of the tuple is the lower bin-edge of the bin.
hist2.bins[1]
h = hist2.bins[1][1]
h.plot.matplotlib()
h.bin_entries()
Explanation: Accessing a sub-histogram
Depending on the histogram type of the first axis, hg.Bin or other, one can access the sub-histograms directly from:
hist.values or 
hist.bins
End of explanation
histy = hg.Categorize('eyeColor')
histx = hg.Categorize('favoriteFruit', value=histy)
histx.fill.numpy(df)
histx.plot.matplotlib();
# show the datatype(s) of the histogram
histx.datatype
Explanation: Histogram types recap
So far we have covered the histogram types: 
- Bin histograms: with a fixed range and even-sized bins,
- SparselyBin histograms: open-ended and with a fixed bin-width,
- CentrallyBin histograms: open-ended and using bin centers.
- IrregularlyBin histograms: open-ended and using (irregular) bin edges,
All of these process numeric variables only.
Categorical variables
For categorical variables use the Categorize histogram
- Categorize histograms: accepting categorical variables such as strings and booleans.
End of explanation
histy = hg.Categorize('isActive')
histy.fill.numpy(df)
histy.plot.matplotlib();
histy.bin_entries()
histy.bin_labels()
# histy.bin_centers() will work as well for Categorize histograms
Explanation: Categorize histograms also accept booleans:
End of explanation
hmin = df.hg_Minimize('latitude')
hmax = df.hg_Maximize('longitude')
(hmin.min, hmax.max)
havg = df.hg_Average('latitude')
hdev = df.hg_Deviate('longitude')
(havg.mean, hdev.mean, hdev.variance)
hsum = df.hg_Sum('age')
hsum.sum
# let's illustrate the Stack histogram with longitude distribution
# first we plot the regular distribution
hl = df.hg_SparselyBin(25, 'longitude')
hl.plot.matplotlib();
# Stack counts how often data points are greater or equal to the provided thresholds 
thresholds = [-200, -150, -100, -50, 0, 50, 100, 150, 200]
hs = df.hg_Stack(thresholds=thresholds, quantity='longitude')
hs.thresholds
hs.bin_entries()
Explanation: Other histogram functionality
There are several more histogram types:
- Minimize, Maximize: keep track of the min or max value of a numeric distribution,
- Average, Deviate: keep track of the mean or mean and standard deviation of a numeric distribution,
- Sum: keep track of the sum of a numeric distribution,
- Stack: keep track how many data points pass certain thresholds.
- Bag: works like a dict, it keeps tracks of all unique values encountered in a column, and can also do this for vector s of numbers. For strings, Bag works just like the Categorize histogram.
End of explanation
hav = hg.Deviate('age')
hlo = hg.SparselyBin(25, 'longitude', value=hav)
hlo.fill.numpy(df)
hlo.bins
hlo.plot.matplotlib();
Explanation: Stack histograms are useful to make efficiency curves.
With all these histograms you can make multi-dimensional histograms. For example, you can evaluate the mean and standard deviation of one feature as a function of bins of another feature. (A "profile" plot, similar to a box plot.)
End of explanation
# For example, call this convenience function to make the same histogram as above:
hlo = df.hg_SparselyProfileErr(25, 'longitude', 'age')
hlo.plot.matplotlib();
Explanation: Convenience functions
There are several convenience functions to make such composed histograms. These are:
- Profile: Convenience function for creating binwise averages.
- SparselyProfile: Convenience function for creating sparsely binned binwise averages.
- ProfileErr: Convenience function for creating binwise averages and variances.
- SparselyProfile: Convenience function for creating sparsely binned binwise averages and variances.
- TwoDimensionallyHistogram: Convenience function for creating a conventional, two-dimensional histogram.
- TwoDimensionallySparselyHistogram: Convenience function for creating a sparsely binned, two-dimensional histogram.
End of explanation
hists = df.hg_make_histograms()
hists.keys()
h = hists['transaction']
h.plot.matplotlib();
h = hists['date']
h.plot.matplotlib();
# you can also select which and make multi-dimensional histograms
hists = df.hg_make_histograms(features = ['longitude:age'])
hist = hists['longitude:age']
hist.plot.matplotlib();
Explanation: Overview of histograms
Here you can find the list of all available histograms and aggregators and how to use each one: 
https://histogrammar.github.io/histogrammar-docs/specification/1.0/
The most useful aggregators are the following. Tinker with them to get familiar; building up an analysis is easier when you know "there's an app for that."
Simple counters:
Count: just counts. Every aggregator has an entries field, but Count only has this field.
Average and Deviate: add mean and variance, cumulatively.
Minimize and Maximize: lowest and highest value seen.
Histogram-like objects:
Bin and SparselyBin: split a numerical domain into uniform bins and redirect aggregation into those bins.
Categorize: split a string-valued domain by unique values; good for making bar charts (which are histograms with a string-valued axis).
CentrallyBin and IrregularlyBin: split a numerical domain into arbitrary subintervals, usually for separate plots like particle pseudorapidity or collision centrality.
Collections:
Label, UntypedLabel, and Index: bundle objects with string-based keys (Label and UntypedLabel) or simply an ordered array (effectively, integer-based keys) consisting of a single type (Label and Index) or any types (UntypedLabel).
Branch: for the fourth case, an ordered array of any types. A Branch is useful as a "cable splitter". For instance, to make a histogram that tracks minimum and maximum value, do this:
Making many histograms at once
There a nice method to make many histograms in one go. See here.
By default automagical binning is applied to make the histograms.
More details one how to use this function are found in in the advanced tutorial.
End of explanation
# storage
hist.toJsonFile('long_age.json')
# retrieval
factory = hg.Factory()
hist2 = factory.fromJsonFile('long_age.json')
hist2.plot.matplotlib();
# we can store the histograms if we want to
import json
from histogrammar.util import dumper
# store
with open('histograms.json', 'w') as outfile:
    json.dump(hists, outfile, default=dumper)
# and load again
with open('histograms.json') as handle:
    hists2 = json.load(handle)
hists.keys()
Explanation: Storage
Histograms can be easily stored and retrieved in/from the json format.
End of explanation
<END_TASK>