It Takes Two to Tango: Navigating Conceptualizations of NLP Tasks and
  Measurements of Performance
Arjun Subramonian Xingdi Yuan Hal Daumé III Su Lin Blodgett
Progress in NLP is increasingly measured through benchmarks; hence,
contextualizing progress requires understanding when and why practitioners may
disagree about the validity of benchmarks. We develop a taxonomy of
disagreement, drawing on tools from measurement modeling, and distinguish
between two types of disagreement: 1) how tasks are conceptualized and 2) how
measurements of model performance are operationalized. To provide evidence for
our taxonomy, we conduct a meta-analysis of relevant literature to understand
how NLP tasks are conceptualized, as well as a survey of practitioners about
their impressions of different factors that affect benchmark validity. Our
meta-analysis and survey across eight tasks, ranging from coreference
resolution to question answering, uncover that tasks are generally not clearly
and consistently conceptualized and benchmarks suffer from operationalization
disagreements. These findings support our proposed taxonomy of disagreement.
Finally, based on our taxonomy, we present a framework for constructing
benchmarks and documenting their limitations.