Automated Annotation with Generative AI Requires Validation
Nicholas Pangakis Samuel Wolken Neil Fasching
Generative large language models (LLMs) can be a powerful tool for augmenting
text annotation procedures, but their performance varies across annotation
tasks due to prompt quality, text data idiosyncrasies, and conceptual
difficulty. Because these challenges will persist even as LLM technology
improves, we argue that any automated annotation process using an LLM must
validate the LLM's performance against labels generated by humans. To this end,
we outline a workflow to harness the annotation potential of LLMs in a
principled, efficient way. Using GPT-4, we validate this approach by
replicating 27 annotation tasks across 11 datasets from recent social science
articles in high-impact journals. We find that LLM performance for text
annotation is promising but highly contingent on both the dataset and the type
of annotation task, which reinforces the necessity to validate on a
task-by-task basis. We make available easy-to-use software designed to
implement our workflow and streamline the deployment of LLMs for automated
annotation.