ACTOR: Active Learning with Annotator-specific Classification Heads to
  Embrace Human Label Variation
Xinpeng Wang Barbara Plank
Label aggregation such as majority voting is commonly used to resolve
annotator disagreement in dataset creation. However, this may disregard
minority values and opinions. Recent studies indicate that learning from
individual annotations outperforms learning from aggregated labels, though they
require a considerable amount of annotation. Active learning, as an annotation
cost-saving strategy, has not been fully explored in the context of learning
from disagreement. We show that in the active learning setting, a multi-head
model performs significantly better than a single-head model in terms of
uncertainty estimation. By designing and evaluating acquisition functions with
annotator-specific heads on two datasets, we show that group-level entropy
works generally well on both datasets. Importantly, it achieves performance in
terms of both prediction and uncertainty estimation comparable to full-scale
training from disagreement, while saving up to 70% of the annotation budget.