Multi-scale Time-stepping of Partial Differential Equations with
  Transformers
AmirPouya Hemmasian Amir Barati Farimani
Developing fast surrogates for Partial Differential Equations (PDEs) will
accelerate design and optimization in almost all scientific and engineering
applications. Neural networks have been receiving ever-increasing attention and
demonstrated remarkable success in computational modeling of PDEs, however;
their prediction accuracy is not at the level of full deployment. In this work,
we utilize the transformer architecture, the backbone of numerous
state-of-the-art AI models, to learn the dynamics of physical systems as the
mixing of spatial patterns learned by a convolutional autoencoder. Moreover, we
incorporate the idea of multi-scale hierarchical time-stepping to increase the
prediction speed and decrease accumulated error over time. Our model achieves
similar or better results in predicting the time-evolution of Navier-Stokes
equations compared to the powerful Fourier Neural Operator (FNO) and two
transformer-based neural operators OFormer and Galerkin Transformer.