Dozerformer: Sequence Adaptive Sparse Transformer for Multivariate Time
  Series Forecasting
Yifan Zhang Rui Wu Sergiu M. Dascalu Frederick C. Harris Jr
Transformers have achieved remarkable performance in multivariate time
series(MTS) forecasting due to their capability to capture long-term
dependencies. However, the canonical attention mechanism has two key
limitations: (1) its quadratic time complexity limits the sequence length, and
(2) it generates future values from the entire historical sequence. To address
this, we propose a Dozer Attention mechanism consisting of three sparse
components: (1) Local, each query exclusively attends to keys within a
localized window of neighboring time steps. (2) Stride, enables each query to
attend to keys at predefined intervals. (3) Vary, allows queries to selectively
attend to keys from a subset of the historical sequence. Notably, the size of
this subset dynamically expands as forecasting horizons extend. Those three
components are designed to capture essential attributes of MTS data, including
locality, seasonality, and global temporal dependencies. Additionally, we
present the Dozerformer Framework, incorporating the Dozer Attention mechanism
for the MTS forecasting task. We evaluated the proposed Dozerformer framework
with recent state-of-the-art methods on nine benchmark datasets and confirmed
its superior performance. The code will be released after the manuscript is
accepted.