Once Upon a $\textit{Time}$ in $\textit{Graph}$: Relative-Time
  Pretraining for Complex Temporal Reasoning
Sen Yang Xin Li Lidong Bing Wai Lam
Our physical world is constantly evolving over time, rendering challenges for
pre-trained language models to understand and reason over the temporal contexts
of texts. Existing work focuses on strengthening the direct association between
a piece of text and its time-stamp. However, the knowledge-time association is
usually insufficient for the downstream tasks that require reasoning over
temporal dependencies between knowledge. In this work, we make use of the
underlying nature of time, all temporally-scoped sentences are strung together
through a one-dimensional time axis, and suggest creating a graph structure
based on the relative placements of events along the time axis. Inspired by the
graph view, we propose RemeMo ($\underline{Re}$lative Ti$\underline{me}$
$\underline{Mo}$deling), which explicitly connects all temporally-scoped facts
by modeling the time relations between any two sentences. Experimental results
show that RemeMo outperforms the baseline T5 on multiple temporal question
answering datasets under various settings. Further analysis suggests that
RemeMo is especially good at modeling long-range complex temporal dependencies.
We release our code and pre-trained checkpoints at
$\href{https://github.com/DAMO-NLP-SG/RemeMo}{\text{this url}}$.