Assessing Distractors in Multiple-Choice Tests
Vatsal Raina Adian Liusie Mark Gales
Multiple-choice tests are a common approach for assessing candidates'
comprehension skills. Standard multiple-choice reading comprehension exams
require candidates to select the correct answer option from a discrete set
based on a question in relation to a contextual passage. For appropriate
assessment, the distractor answer options must by definition be incorrect but
plausible and diverse. However, generating good quality distractors satisfying
these criteria is a challenging task for content creators. We propose automated
assessment metrics for the quality of distractors in multiple-choice reading
comprehension tests. Specifically, we define quality in terms of the
incorrectness, plausibility and diversity of the distractor options. We assess
incorrectness using the classification ability of a binary multiple-choice
reading comprehension system. Plausibility is assessed by considering the
distractor confidence - the probability mass associated with the distractor
options for a standard multi-class multiple-choice reading comprehension
system. Diversity is assessed by pairwise comparison of an embedding-based
equivalence metric between the distractors of a question. To further validate
the plausibility metric we compare against candidate distributions over
multiple-choice questions and agreement with a ChatGPT model's interpretation
of distractor plausibility and diversity.