PT-Tuning: Bridging the Gap between Time Series Masked Reconstruction
  and Forecasting via Prompt Token Tuning
Hao Liu Jinrui Gan Xiaoxuan Fan Yi Zhang Chuanxian Luo Jing Zhang Guangxin Jiang Yucheng Qian Changwei Zhao Huan Ma Zhenyu Guo
Self-supervised learning has been actively studied in time series domain
recently, especially for masked reconstruction. Most of these methods follow
the "Pre-training + Fine-tuning" paradigm in which a new decoder replaces the
pre-trained decoder to fit for a specific downstream task, leading to
inconsistency of upstream and downstream tasks. In this paper, we first point
out that the unification of task objectives and adaptation for task difficulty
are critical for bridging the gap between time series masked reconstruction and
forecasting. By reserving the pre-trained mask token during fine-tuning stage,
the forecasting task can be taken as a special case of masked reconstruction,
where the future values are masked and reconstructed based on history values.
It guarantees the consistency of task objectives but there is still a gap in
task difficulty. Because masked reconstruction can utilize contextual
information while forecasting can only use historical information to
reconstruct. To further mitigate the existed gap, we propose a simple yet
effective prompt token tuning (PT-Tuning) paradigm, in which all pre-trained
parameters are frozen and only a few trainable prompt tokens are added to
extended mask tokens in element-wise manner. Extensive experiments on
real-world datasets demonstrate the superiority of our proposed paradigm with
state-of-the-art performance compared to representation learning and end-to-end
supervised forecasting methods.