A Survey of Deep Learning and Foundation Models for Time Series
  Forecasting
John A. Miller Mohammed Aldosari Farah Saeed Nasid Habib Barna Subas Rana I. Budak Arpinar Ninghao Liu
Deep Learning has been successfully applied to many application domains, yet
its advantages have been slow to emerge for time series forecasting. For
example, in the well-known Makridakis (M) Competitions, hybrids of traditional
statistical or machine learning techniques have only recently become the top
performers. With the recent architectural advances in deep learning being
applied to time series forecasting (e.g., encoder-decoders with attention,
transformers, and graph neural networks), deep learning has begun to show
significant advantages. Still, in the area of pandemic prediction, there remain
challenges for deep learning models: the time series is not long enough for
effective training, unawareness of accumulated scientific knowledge, and
interpretability of the model. To this end, the development of foundation
models (large deep learning models with extensive pre-training) allows models
to understand patterns and acquire knowledge that can be applied to new related
problems before extensive training data becomes available. Furthermore, there
is a vast amount of knowledge available that deep learning models can tap into,
including Knowledge Graphs and Large Language Models fine-tuned with scientific
domain knowledge. There is ongoing research examining how to utilize or inject
such knowledge into deep learning models. In this survey, several
state-of-the-art modeling techniques are reviewed, and suggestions for further
work are provided.