Sparse-VQ Transformer: An FFN-Free Framework with Vector Quantization
  for Enhanced Time Series Forecasting
Yanjun Zhao Tian Zhou Chao Chen Liang Sun Yi Qian Rong Jin
Time series analysis is vital for numerous applications, and transformers
have become increasingly prominent in this domain. Leading methods customize
the transformer architecture from NLP and CV, utilizing a patching technique to
convert continuous signals into segments. Yet, time series data are uniquely
challenging due to significant distribution shifts and intrinsic noise levels.
To address these two challenges,we introduce the Sparse Vector Quantized
FFN-Free Transformer (Sparse-VQ). Our methodology capitalizes on a sparse
vector quantization technique coupled with Reverse Instance Normalization
(RevIN) to reduce noise impact and capture sufficient statistics for
forecasting, serving as an alternative to the Feed-Forward layer (FFN) in the
transformer architecture. Our FFN-free approach trims the parameter count,
enhancing computational efficiency and reducing overfitting. Through
evaluations across ten benchmark datasets, including the newly introduced CAISO
dataset, Sparse-VQ surpasses leading models with a 7.84% and 4.17% decrease in
MAE for univariate and multivariate time series forecasting, respectively.
Moreover, it can be seamlessly integrated with existing transformer-based
models to elevate their performance.