Sparse Binary Transformers for Multivariate Time Series Modeling
Matt Gorbett Hossein Shirazi Indrakshi Ray
Compressed Neural Networks have the potential to enable deep learning across
new applications and smaller computational environments. However, understanding
the range of learning tasks in which such models can succeed is not well
studied. In this work, we apply sparse and binary-weighted Transformers to
multivariate time series problems, showing that the lightweight models achieve
accuracy comparable to that of dense floating-point Transformers of the same
structure. Our model achieves favorable results across three time series
learning tasks: classification, anomaly detection, and single-step forecasting.
Additionally, to reduce the computational complexity of the attention
mechanism, we apply two modifications, which show little to no decline in model
performance: 1) in the classification task, we apply a fixed mask to the query,
key, and value activations, and 2) for forecasting and anomaly detection, which
rely on predicting outputs at a single point in time, we propose an attention
mask to allow computation only at the current time step. Together, each
compression technique and attention modification substantially reduces the
number of non-zero operations necessary in the Transformer. We measure the
computational savings of our approach over a range of metrics including
parameter count, bit size, and floating point operation (FLOPs) count, showing
up to a 53x reduction in storage size and up to 10.5x reduction in FLOPs.