<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 Rationale
For some certain loss functions, such the the negative evidence lower bound (NELBO) in variational inference, they are generally analytically intractable and thus unavailable in closed-form. As such, we might need to resort to taking stochastic estimates of the loss function. In these situations, it is very important to study and understand the robustness of the estimations we are making, particularly in terms of bias and variance. When proposing a new estimator, we may be interested in evaluating the loss at a fined-grained level - not only per batch, but perhaps even per data-point. 
This notebook explores storing the recorded losses in Pandas Dataframes. The recorded losses are 3d, with dimensions corresponding to epochs, batches, and data-points. Specifically, they are of shape (n_epochs, n_batches, batch_size). Instead of using the deprecated Panel functionality from Pandas, we explore the preferred MultiIndex Dataframe.
Lastly, we play around with various data serialization formats supported out-of-the-box by Pandas. This might be useful if the training is GPU-intensive, so the script runs and records the loss remotely on a supercomputer, and we must write the results to file, download them and finally analyze them locally. This is usually trivial, but it is unclear what the behaviour is for more complex MultiIndex dataframes. We restrict our attention to the CSV format, which is human-friendly but very slow and inefficient, and the HDF5, which is basically diametrically opposed - it's basically completely inscrutable, but is very fast and takes up laess space.
Synthetic Data
Step1:  MultiIndex Dataframe
Step2:  Visualization
In this contrived scenario, loss2 is more biased and has higher variance.
Step3:  CSV Read/Write
Step4:  HDF5 Read/Write
HDF5 writing is orders of magnitude faster.
Step5:  Furthermore, the file sizes are significantly smaller.
Step6:  Lastly, it is far more numerical precise.
<ASSISTANT_TASK:>
Python Code:
# create some noise
a = np.random.randn(50, 600, 100)
a.shape
# create some noise with higher variance and add bias.
b = 2. * np.random.randn(*a.shape) + 1.
b.shape
# manufacture some loss function
# there are n_epochs * n_batchs * batch_size 
# recorded values of the loss
loss = 10 / np.linspace(1, 100, a.size)
loss.shape
Explanation: Rationale
For some certain loss functions, such the the negative evidence lower bound (NELBO) in variational inference, they are generally analytically intractable and thus unavailable in closed-form. As such, we might need to resort to taking stochastic estimates of the loss function. In these situations, it is very important to study and understand the robustness of the estimations we are making, particularly in terms of bias and variance. When proposing a new estimator, we may be interested in evaluating the loss at a fined-grained level - not only per batch, but perhaps even per data-point. 
This notebook explores storing the recorded losses in Pandas Dataframes. The recorded losses are 3d, with dimensions corresponding to epochs, batches, and data-points. Specifically, they are of shape (n_epochs, n_batches, batch_size). Instead of using the deprecated Panel functionality from Pandas, we explore the preferred MultiIndex Dataframe.
Lastly, we play around with various data serialization formats supported out-of-the-box by Pandas. This might be useful if the training is GPU-intensive, so the script runs and records the loss remotely on a supercomputer, and we must write the results to file, download them and finally analyze them locally. This is usually trivial, but it is unclear what the behaviour is for more complex MultiIndex dataframes. We restrict our attention to the CSV format, which is human-friendly but very slow and inefficient, and the HDF5, which is basically diametrically opposed - it's basically completely inscrutable, but is very fast and takes up laess space.
Synthetic Data
End of explanation
# we will create the indices from the 
# product of these iterators
list(map(range, a.shape))
# create the MultiIndex
index = pd.MultiIndex.from_product(
    list(map(range, a.shape)), 
    names=['epoch', 'batch', 'datapoint']
)
# create the dataframe that records the two losses
df = pd.DataFrame(
    dict(loss1=loss+np.ravel(a), 
         loss2=loss+np.ravel(b)), 
    index=index
)
df
Explanation: MultiIndex Dataframe
End of explanation
# some basic plotting
fig, ax = plt.subplots()
df.groupby(['epoch', 'batch']).mean().plot(ax=ax)
plt.show()
Explanation: Visualization
In this contrived scenario, loss2 is more biased and has higher variance.
End of explanation
%%time
df.to_csv('losses.csv')
!ls -lh losses.csv
%%time
df_from_csv = pd.read_csv('losses.csv', index_col=['epoch', 'batch', 'datapoint'], float_precision='high')
# does not recover exactly due to insufficient floating point precision
df_from_csv.equals(df)
# but it has recovered it up to some tiny epsilon
((df-df_from_csv)**2 < 1e-25).all()
Explanation: CSV Read/Write
End of explanation
%%time
df.to_hdf('store.h5', key='losses')
Explanation: HDF5 Read/Write
HDF5 writing is orders of magnitude faster.
End of explanation
!ls -lh store.h5
%%time
df_from_hdf = pd.read_hdf('store.h5', key='losses')
Explanation: Furthermore, the file sizes are significantly smaller.
End of explanation
df.equals(df_from_hdf)
Explanation: Lastly, it is far more numerical precise.
End of explanation
<END_TASK>