Improving Position Encoding of Transformers for Multivariate Time Series
  Classification
Navid Mohammadi Foumani Chang Wei Tan Geoffrey I. Webb Mahsa Salehi
Transformers have demonstrated outstanding performance in many applications
of deep learning. When applied to time series data, transformers require
effective position encoding to capture the ordering of the time series data.
The efficacy of position encoding in time series analysis is not well-studied
and remains controversial, e.g., whether it is better to inject absolute
position encoding or relative position encoding, or a combination of them. In
order to clarify this, we first review existing absolute and relative position
encoding methods when applied in time series classification. We then proposed a
new absolute position encoding method dedicated to time series data called time
Absolute Position Encoding (tAPE). Our new method incorporates the series
length and input embedding dimension in absolute position encoding.
Additionally, we propose computationally Efficient implementation of Relative
Position Encoding (eRPE) to improve generalisability for time series. We then
propose a novel multivariate time series classification (MTSC) model combining
tAPE/eRPE and convolution-based input encoding named ConvTran to improve the
position and data embedding of time series data. The proposed absolute and
relative position encoding methods are simple and efficient. They can be easily
integrated into transformer blocks and used for downstream tasks such as
forecasting, extrinsic regression, and anomaly detection. Extensive experiments
on 32 multivariate time-series datasets show that our model is significantly
more accurate than state-of-the-art convolution and transformer-based models.
Code and models are open-sourced at
\url{https://github.com/Navidfoumani/ConvTran}.