Attention as Robust Representation for Time Series Forecasting
PeiSong Niu Tian Zhou Xue Wang Liang Sun Rong Jin
Time series forecasting is essential for many practical applications, with
the adoption of transformer-based models on the rise due to their impressive
performance in NLP and CV. Transformers' key feature, the attention mechanism,
dynamically fusing embeddings to enhance data representation, often relegating
attention weights to a byproduct role. Yet, time series data, characterized by
noise and non-stationarity, poses significant forecasting challenges. Our
approach elevates attention weights as the primary representation for time
series, capitalizing on the temporal relationships among data points to improve
forecasting accuracy. Our study shows that an attention map, structured using
global landmarks and local windows, acts as a robust kernel representation for
data points, withstanding noise and shifts in distribution. Our method
outperforms state-of-the-art models, reducing mean squared error (MSE) in
multivariate time series forecasting by a notable 3.6% without altering the
core neural network architecture. It serves as a versatile component that can
readily replace recent patching based embedding schemes in transformer-based
models, boosting their performance.