A Survey on Time-Series Pre-Trained Models
Qianli Ma Zhen Liu Zhenjing Zheng Ziyang Huang Siying Zhu Zhongzhong Yu James T. Kwok
Time-Series Mining (TSM) is an important research area since it shows great
potential in practical applications. Deep learning models that rely on massive
labeled data have been utilized for TSM successfully. However, constructing a
large-scale well-labeled dataset is difficult due to data annotation costs.
Recently, Pre-Trained Models have gradually attracted attention in the time
series domain due to their remarkable performance in computer vision and
natural language processing. In this survey, we provide a comprehensive review
of Time-Series Pre-Trained Models (TS-PTMs), aiming to guide the understanding,
applying, and studying TS-PTMs. Specifically, we first briefly introduce the
typical deep learning models employed in TSM. Then, we give an overview of
TS-PTMs according to the pre-training techniques. The main categories we
explore include supervised, unsupervised, and self-supervised TS-PTMs. Further,
extensive experiments are conducted to analyze the advantages and disadvantages
of transfer learning strategies, Transformer-based models, and representative
TS-PTMs. Finally, we point out some potential directions of TS-PTMs for future
work.