Probing the limit of hydrologic predictability with the Transformer
  network
Jiangtao Liu Yuchen Bian Chaopeng Shen
For a number of years since its introduction to hydrology, recurrent neural
networks like long short-term memory (LSTM) have proven remarkably difficult to
surpass in terms of daily hydrograph metrics on known, comparable benchmarks.
Outside of hydrology, Transformers have now become the model of choice for
sequential prediction tasks, making it a curious architecture to investigate.
Here, we first show that a vanilla Transformer architecture is not competitive
against LSTM on the widely benchmarked CAMELS dataset, and lagged especially
for the high-flow metrics due to short-term processes. However, a
recurrence-free variant of Transformer can obtain mixed comparisons with LSTM,
producing the same Kling-Gupta efficiency coefficient (KGE), along with other
metrics. The lack of advantages for the Transformer is linked to the Markovian
nature of the hydrologic prediction problem. Similar to LSTM, the Transformer
can also merge multiple forcing dataset to improve model performance. While the
Transformer results are not higher than current state-of-the-art, we still
learned some valuable lessons: (1) the vanilla Transformer architecture is not
suitable for hydrologic modeling; (2) the proposed recurrence-free modification
can improve Transformer performance so future work can continue to test more of
such modifications; and (3) the prediction limits on the dataset should be
close to the current state-of-the-art model. As a non-recurrent model, the
Transformer may bear scale advantages for learning from bigger datasets and
storing knowledge. This work serves as a reference point for future
modifications of the model.