On Degrees of Freedom in Defining and Testing Natural Language
  Understanding
Saku Sugawara Shun Tsugita
Natural language understanding (NLU) studies often exaggerate or
underestimate the capabilities of systems, thereby limiting the reproducibility
of their findings. These erroneous evaluations can be attributed to the
difficulty of defining and testing NLU adequately. In this position paper, we
reconsider this challenge by identifying two types of researcher degrees of
freedom. We revisit Turing's original interpretation of the Turing test and
indicate that an NLU test does not provide an operational definition; it merely
provides inductive evidence that the test subject understands the language
sufficiently well to meet stakeholder objectives. In other words, stakeholders
are free to arbitrarily define NLU through their objectives. To use the test
results as inductive evidence, stakeholders must carefully assess if the
interpretation of test scores is valid or not. However, designing and using NLU
tests involve other degrees of freedom, such as specifying target skills and
defining evaluation metrics. As a result, achieving consensus among
stakeholders becomes difficult. To resolve this issue, we propose a validity
argument, which is a framework comprising a series of validation criteria
across test components. By demonstrating that current practices in NLU studies
can be associated with those criteria and organizing them into a comprehensive
checklist, we prove that the validity argument can serve as a coherent
guideline for designing credible test sets and facilitating scientific
communication.