Conformer: Embedding Continuous Attention in Vision Transformer for
  Weather Forecasting
Hira Saleem Flora Salim Cormac Purcell
Operational weather forecasting system relies on computationally expensive
physics-based models. Although Transformers-based models have shown remarkable
potential in weather forecasting, Transformers are discrete models which limit
their ability to learn the continuous spatio-temporal features of the dynamical
weather system. We address this issue with Conformer, a spatio-temporal
Continuous Vision Transformer for weather forecasting. Conformer is designed to
learn the continuous weather evolution over time by implementing continuity in
the multi-head attention mechanism. The attention mechanism is encoded as a
differentiable function in the transformer architecture to model the complex
weather dynamics. We evaluate Conformer against a state-of-the-art Numerical
Weather Prediction (NWP) model and several deep learning based weather
forecasting models. Conformer outperforms some of the existing data-driven
models at all lead times while only being trained at lower resolution data.