Don't Blame the Data, Blame the Model: Understanding Noise and Bias When
  Learning from Subjective Annotations
Abhishek Anand Negar Mokhberian Prathyusha Naresh Kumar Anweasha Saha Zihao He Ashwin Rao Fred Morstatter Kristina Lerman
Researchers have raised awareness about the harms of aggregating labels
especially in subjective tasks that naturally contain disagreements among human
annotators. In this work we show that models that are only provided aggregated
labels show low confidence on high-disagreement data instances. While previous
studies consider such instances as mislabeled, we argue that the reason the
high-disagreement text instances have been hard-to-learn is that the
conventional aggregated models underperform in extracting useful signals from
subjective tasks. Inspired by recent studies demonstrating the effectiveness of
learning from raw annotations, we investigate classifying using Multiple Ground
Truth (Multi-GT) approaches. Our experiments show an improvement of confidence
for the high-disagreement instances.