STanHop: Sparse Tandem Hopfield Model for Memory-Enhanced Time Series
  Prediction
Dennis Wu Jerry Yao-Chieh Hu Weijian Li Bo-Yu Chen Han Liu
We present STanHop-Net (Sparse Tandem Hopfield Network) for multivariate time
series prediction with memory-enhanced capabilities. At the heart of our
approach is STanHop, a novel Hopfield-based neural network block, which
sparsely learns and stores both temporal and cross-series representations in a
data-dependent fashion. In essence, STanHop sequentially learn temporal
representation and cross-series representation using two tandem sparse Hopfield
layers. In addition, StanHop incorporates two additional external memory
modules: a Plug-and-Play module and a Tune-and-Play module for train-less and
task-aware memory-enhancements, respectively. They allow StanHop-Net to swiftly
respond to certain sudden events. Methodologically, we construct the
StanHop-Net by stacking STanHop blocks in a hierarchical fashion, enabling
multi-resolution feature extraction with resolution-specific sparsity.
Theoretically, we introduce a sparse extension of the modern Hopfield model
(Generalized Sparse Modern Hopfield Model) and show that it endows a tighter
memory retrieval error compared to the dense counterpart without sacrificing
memory capacity. Empirically, we validate the efficacy of our framework on both
synthetic and real-world settings.