Generalizable Error Modeling for Search Relevance Data Annotation Tasks
Heinrich Peters Alireza Hashemi James Rae
Human data annotation is critical in shaping the quality of machine learning
(ML) and artificial intelligence (AI) systems. One significant challenge in
this context is posed by annotation errors, as their effects can degrade the
performance of ML models. This paper presents a predictive error model trained
to detect potential errors in search relevance annotation tasks for three
industry-scale ML applications (music streaming, video streaming, and mobile
apps) and assesses its potential to enhance the quality and efficiency of the
data annotation process. Drawing on real-world data from an extensive search
relevance annotation program, we illustrate that errors can be predicted with
moderate model performance (AUC=0.65-0.75) and that model performance
generalizes well across applications (i.e., a global, task-agnostic model
performs on par with task-specific models). We present model explainability
analyses to identify which types of features are the main drivers of predictive
performance. Additionally, we demonstrate the usefulness of the model in the
context of auditing, where prioritizing tasks with high predicted error
probabilities considerably increases the amount of corrected annotation errors
(e.g., 40% efficiency gains for the music streaming application). These results
underscore that automated error detection models can yield considerable
improvements in the efficiency and quality of data annotation processes. Thus,
our findings reveal critical insights into effective error management in the
data annotation process, thereby contributing to the broader field of
human-in-the-loop ML.