A novel interface for adversarial trivia question-writing
Jason Liu
A critical component when developing question-answering AIs is an adversarial
dataset that challenges models to adapt to the complex syntax and reasoning
underlying our natural language. Present techniques for procedurally generating
adversarial texts are not robust enough for training on complex tasks such as
answering multi-sentence trivia questions. We instead turn to human-generated
data by introducing an interface for collecting adversarial human-written
trivia questions. Our interface is aimed towards question writers and players
of Quiz Bowl, a buzzer-based trivia competition where paragraph-long questions
consist of a sequence of clues of decreasing difficulty. To incentivize usage,
a suite of machine learning-based tools in our interface assist humans in
writing questions that are more challenging to answer for Quiz Bowl players and
computers alike. Not only does our interface gather training data for the
groundbreaking Quiz Bowl AI project QANTA, but it is also a proof-of-concept of
future adversarial data collection for question-answering systems. The results
of performance-testing our interface with ten originally-composed questions
indicate that, despite some flaws, our interface's novel question-writing
features as well as its real-time exposure of useful responses from our machine
models could facilitate and enhance the collection of adversarial questions.