The Road to Quality is Paved with Good Revisions: A Detailed Evaluation
  Methodology for Revision Policies in Incremental Sequence Labelling
Brielen Madureira Patrick Kahardipraja David Schlangen
Incremental dialogue model components produce a sequence of output prefixes
based on incoming input. Mistakes can occur due to local ambiguities or to
wrong hypotheses, making the ability to revise past outputs a desirable
property that can be governed by a policy. In this work, we formalise and
characterise edits and revisions in incremental sequence labelling and propose
metrics to evaluate revision policies. We then apply our methodology to profile
the incremental behaviour of three Transformer-based encoders in various tasks,
paving the road for better revision policies.