TIMELINE: Exhaustive Annotation of Temporal Relations Supporting the
  Automatic Ordering of Events in News Articles
Sarah Alsayyahi Riza Batista-Navarro
Temporal relation extraction models have thus far been hindered by a number
of issues in existing temporal relation-annotated news datasets, including: (1)
low inter-annotator agreement due to the lack of specificity of their
annotation guidelines in terms of what counts as a temporal relation; (2) the
exclusion of long-distance relations within a given document (those spanning
across different paragraphs); and (3) the exclusion of events that are not
centred on verbs. This paper aims to alleviate these issues by presenting a new
annotation scheme that clearly defines the criteria based on which temporal
relations should be annotated. Additionally, the scheme includes events even if
they are not expressed as verbs (e.g., nominalised events). Furthermore, we
propose a method for annotating all temporal relations -- including
long-distance ones -- which automates the process, hence reducing time and
manual effort on the part of annotators. The result is a new dataset, the
TIMELINE corpus, in which improved inter-annotator agreement was obtained, in
comparison with previously reported temporal relation datasets. We report the
results of training and evaluating baseline temporal relation extraction models
on the new corpus, and compare them with results obtained on the widely used
MATRES corpus.