If in a Crowdsourced Data Annotation Pipeline, a GPT-4
Zeyu He Chieh-Yang Huang Chien-Kuang Cornelia Ding Shaurya Rohatgi Ting-Hao 'Kenneth' Huang
Recent studies indicated GPT-4 outperforms online crowd workers in data
labeling accuracy, notably workers from Amazon Mechanical Turk (MTurk).
However, these studies were criticized for deviating from standard
crowdsourcing practices and emphasizing individual workers' performances over
the whole data-annotation process. This paper compared GPT-4 and an ethical and
well-executed MTurk pipeline, with 415 workers labeling 3,177 sentence segments
from 200 scholarly articles using the CODA-19 scheme. Two worker interfaces
yielded 127,080 labels, which were then used to infer the final labels through
eight label-aggregation algorithms. Our evaluation showed that despite best
practices, MTurk pipeline's highest accuracy was 81.5%, whereas GPT-4 achieved
83.6%. Interestingly, when combining GPT-4's labels with crowd labels collected
via an advanced worker interface for aggregation, 2 out of the 8 algorithms
achieved an even higher accuracy (87.5%, 87.0%). Further analysis suggested
that, when the crowd's and GPT-4's labeling strengths are complementary,
aggregating them could increase labeling accuracy.