CoAnnotating: Uncertainty-Guided Work Allocation between Human and Large
  Language Models for Data Annotation
Minzhi Li Taiwei Shi Caleb Ziems Min-Yen Kan Nancy F. Chen Zhengyuan Liu Diyi Yang
Annotated data plays a critical role in Natural Language Processing (NLP) in
training models and evaluating their performance. Given recent developments in
Large Language Models (LLMs), models such as ChatGPT demonstrate zero-shot
capability on many text-annotation tasks, comparable with or even exceeding
human annotators. Such LLMs can serve as alternatives for manual annotation,
due to lower costs and higher scalability. However, limited work has leveraged
LLMs as complementary annotators, nor explored how annotation work is best
allocated among humans and LLMs to achieve both quality and cost objectives. We
propose CoAnnotating, a novel paradigm for Human-LLM co-annotation of
unstructured texts at scale. Under this framework, we utilize uncertainty to
estimate LLMs' annotation capability. Our empirical study shows CoAnnotating to
be an effective means to allocate work from results on different datasets, with
up to 21% performance improvement over random baseline. For code
implementation, see https://github.com/SALT-NLP/CoAnnotating.