Generative Pretrained Hierarchical Transformer for Time Series
  Forecasting
Zhiding Liu Jiqian Yang Mingyue Cheng Yucong Luo Zhi Li
Recent efforts have been dedicated to enhancing time series forecasting
accuracy by introducing advanced network architectures and self-supervised
pretraining strategies. Nevertheless, existing approaches still exhibit two
critical drawbacks. Firstly, these methods often rely on a single dataset for
training, limiting the model's generalizability due to the restricted scale of
the training data. Secondly, the one-step generation schema is widely followed,
which necessitates a customized forecasting head and overlooks the temporal
dependencies in the output series, and also leads to increased training costs
under different horizon length settings.
  To address these issues, we propose a novel generative pretrained
hierarchical transformer architecture for forecasting, named GPHT. There are
two aspects of key designs in GPHT. On the one hand, we advocate for
constructing a mixed dataset for pretraining our model, comprising various
datasets from diverse data scenarios. This approach significantly expands the
scale of training data, allowing our model to uncover commonalities in time
series data and facilitating improved transfer to specific datasets. On the
other hand, GPHT employs an auto-regressive forecasting approach under the
channel-independent assumption, effectively modeling temporal dependencies in
the output series. Importantly, no customized forecasting head is required,
enabling a single model to forecast at arbitrary horizon settings. We conduct
sufficient experiments on eight datasets with mainstream self-supervised
pretraining models and supervised models. The results demonstrated that GPHT
surpasses the baseline models across various fine-tuning and zero/few-shot
learning settings in the traditional long-term forecasting task, providing
support for verifying the feasibility of pretrained time series large models.