A Comparative Study on Annotation Quality of Crowdsourcing and LLM via
  Label Aggregation
Jiyi Li
Whether Large Language Models (LLMs) can outperform crowdsourcing on the data
annotation task is attracting interest recently. Some works verified this issue
with the average performance of individual crowd workers and LLM workers on
some specific NLP tasks by collecting new datasets. However, on the one hand,
existing datasets for the studies of annotation quality in crowdsourcing are
not yet utilized in such evaluations, which potentially provide reliable
evaluations from a different viewpoint. On the other hand, the quality of these
aggregated labels is crucial because, when utilizing crowdsourcing, the
estimated labels aggregated from multiple crowd labels to the same instances
are the eventually collected labels. Therefore, in this paper, we first
investigate which existing crowdsourcing datasets can be used for a comparative
study and create a benchmark. We then compare the quality between individual
crowd labels and LLM labels and make the evaluations on the aggregated labels.
In addition, we propose a Crowd-LLM hybrid label aggregation method and verify
the performance. We find that adding LLM labels from good LLMs to existing
crowdsourcing datasets can enhance the quality of the aggregated labels of the
datasets, which is also higher than the quality of LLM labels themselves.