Crowd-Certain: Label Aggregation in Crowdsourced and Ensemble Learning
  Classification
Mohammad S. Majdi Jeffrey J. Rodriguez
Crowdsourcing systems have been used to accumulate massive amounts of labeled
data for applications such as computer vision and natural language processing.
However, because crowdsourced labeling is inherently dynamic and uncertain,
developing a technique that can work in most situations is extremely
challenging. In this paper, we introduce Crowd-Certain, a novel approach for
label aggregation in crowdsourced and ensemble learning classification tasks
that offers improved performance and computational efficiency for different
numbers of annotators and a variety of datasets. The proposed method uses the
consistency of the annotators versus a trained classifier to determine a
reliability score for each annotator. Furthermore, Crowd-Certain leverages
predicted probabilities, enabling the reuse of trained classifiers on future
sample data, thereby eliminating the need for recurrent simulation processes
inherent in existing methods. We extensively evaluated our approach against ten
existing techniques across ten different datasets, each labeled by varying
numbers of annotators. The findings demonstrate that Crowd-Certain outperforms
the existing methods (Tao, Sheng, KOS, MACE, MajorityVote, MMSR, Wawa,
Zero-Based Skill, GLAD, and Dawid Skene), in nearly all scenarios, delivering
higher average accuracy, F1 scores, and AUC rates. Additionally, we introduce a
variation of two existing confidence score measurement techniques. Finally we
evaluate these two confidence score techniques using two evaluation metrics:
Expected Calibration Error (ECE) and Brier Score Loss. Our results show that
Crowd-Certain achieves higher Brier Score, and lower ECE across the majority of
the examined datasets, suggesting better calibrated results.