How the Advent of Ubiquitous Large Language Models both Stymie and
  Turbocharge Dynamic Adversarial Question Generation
Yoo Yeon Sung Ishani Mondal Jordan Boyd-Graber
Dynamic adversarial question generation, where humans write examples to stump
a model, aims to create examples that are realistic and informative. However,
the advent of large language models (LLMs) has been a double-edged sword for
human authors: more people are interested in seeing and pushing the limits of
these models, but because the models are so much stronger an opponent, they are
harder to defeat. To understand how these models impact adversarial question
writing process, we enrich the writing guidance with LLMs and retrieval models
for the authors to reason why their questions are not adversarial. While
authors could create interesting, challenging adversarial questions, they
sometimes resort to tricks that result in poor questions that are ambiguous,
subjective, or confusing not just to a computer but also to humans. To address
these issues, we propose new metrics and incentives for eliciting good,
challenging questions and present a new dataset of adversarially authored
questions.