Blueprinting the Future: Automatic Item Categorization using
  Hierarchical Zero-Shot and Few-Shot Classifiers
Ting Wang Keith Stelter Jenn Floyd Thomas O'Neill Nathaniel Hendrix Andrew Bazemore Kevin Rode Warren Newton
In testing industry, precise item categorization is pivotal to align exam
questions with the designated content domains outlined in the assessment
blueprint. Traditional methods either entail manual classification, which is
laborious and error-prone, or utilize machine learning requiring extensive
training data, often leading to model underfit or overfit issues. This study
unveils a novel approach employing the zero-shot and few-shot Generative
Pretrained Transformer (GPT) classifier for hierarchical item categorization,
minimizing the necessity for training data, and instead, leveraging human-like
language descriptions to define categories. Through a structured python
dictionary, the hierarchical nature of examination blueprints is navigated
seamlessly, allowing for a tiered classification of items across multiple
levels. An initial simulation with artificial data demonstrates the efficacy of
this method, achieving an average accuracy of 92.91% measured by the F1 score.
This method was further applied to real exam items from the 2022 In-Training
Examination (ITE) conducted by the American Board of Family Medicine (ABFM),
reclassifying 200 items according to a newly formulated blueprint swiftly in 15
minutes, a task that traditionally could span several days among editors and
physicians. This innovative approach not only drastically cuts down
classification time but also ensures a consistent, principle-driven
categorization, minimizing human biases and discrepancies. The ability to
refine classifications by adjusting definitions adds to its robustness and
sustainability.