Modeling and Analyzing Scorer Preferences in Short-Answer Math Questions
Mengxue Zhang Neil Heffernan Andrew Lan
Automated scoring of student responses to open-ended questions, including
short-answer questions, has great potential to scale to a large number of
responses. Recent approaches for automated scoring rely on supervised learning,
i.e., training classifiers or fine-tuning language models on a small number of
responses with human-provided score labels. However, since scoring is a
subjective process, these human scores are noisy and can be highly variable,
depending on the scorer. In this paper, we investigate a collection of models
that account for the individual preferences and tendencies of each human scorer
in the automated scoring task. We apply these models to a short-answer math
response dataset where each response is scored (often differently) by multiple
different human scorers. We conduct quantitative experiments to show that our
scorer models lead to improved automated scoring accuracy. We also conduct
quantitative experiments and case studies to analyze the individual preferences
and tendencies of scorers. We found that scorers can be grouped into several
obvious clusters, with each cluster having distinct features, and analyzed them
in detail.