Streaming Compression of Scientific Data via weak-SINDy
Benjamin P. Russo M. Paul Laiu Richard Archibald
In this paper a streaming weak-SINDy algorithm is developed specifically for
compressing streaming scientific data. The production of scientific data,
either via simulation or experiments, is undergoing an stage of exponential
growth, which makes data compression important and often necessary for storing
and utilizing large scientific data sets. As opposed to classical ``offline"
compression algorithms that perform compression on a readily available data
set, streaming compression algorithms compress data ``online" while the data
generated from simulation or experiments is still flowing through the system.
This feature makes streaming compression algorithms well-suited for scientific
data compression, where storing the full data set offline is often infeasible.
This work proposes a new streaming compression algorithm, streaming weak-SINDy,
which takes advantage of the underlying data characteristics during
compression. The streaming weak-SINDy algorithm constructs feature matrices and
target vectors in the online stage via a streaming integration method in a
memory efficient manner. The feature matrices and target vectors are then used
in the offline stage to build a model through a regression process that aims to
recover equations that govern the evolution of the data. For compressing
high-dimensional streaming data, we adopt a streaming proper orthogonal
decomposition (POD) process to reduce the data dimension and then use the
streaming weak-SINDy algorithm to compress the temporal data of the POD
expansion. We propose modifications to the streaming weak-SINDy algorithm to
accommodate the dynamically updated POD basis. By combining the built model
from the streaming weak-SINDy algorithm and a small amount of data samples, the
full data flow could be reconstructed accurately at a low memory cost, as shown
in the numerical tests.