Taylorformer: Probabilistic Predictions for Time Series and other
  Processes
Omer Nivron Raghul Parthipan Damon J. Wischik
We propose the Taylorformer for time series and other random processes. Its
two key components are: 1) the LocalTaylor wrapper to learn how and when to use
Taylor series-based approximations for predictions, and 2) the MHA-X attention
block which makes predictions in a way inspired by how Gaussian Processes' mean
predictions are linear smoothings of contextual data. Taylorformer outperforms
the state-of-the-art on several forecasting datasets, including electricity,
oil temperatures and exchange rates with at least 14% improvement in MSE on all
tasks, and better likelihood on 5/6 classic Neural Process tasks such as
meta-learning 1D functions. Taylorformer combines desirable features from the
Neural Process (uncertainty-aware predictions and consistency) and forecasting
(predictive accuracy) literature, two previously distinct bodies.