Unveiling the Multi-Annotation Process: Examining the Influence of
  Annotation Quantity and Instance Difficulty on Model Performance
Pritam Kadasi Mayank Singh
The NLP community has long advocated for the construction of multi-annotator
datasets to better capture the nuances of language interpretation,
subjectivity, and ambiguity. This paper conducts a retrospective study to show
how performance scores can vary when a dataset expands from a single annotation
per instance to multiple annotations. We propose a novel multi-annotator
simulation process to generate datasets with varying annotation budgets. We
show that similar datasets with the same annotation budget can lead to varying
performance gains. Our findings challenge the popular belief that models
trained on multi-annotation examples always lead to better performance than
models trained on single or few-annotation examples.