Architectural Sweet Spots for Modeling Human Label Variation by the
  Example of Argument Quality: It's Best to Relate Perspectives!
Philipp Heinisch Matthias Orlikowski Julia Romberg Philipp Cimiano
Many annotation tasks in natural language processing are highly subjective in
that there can be different valid and justified perspectives on what is a
proper label for a given example. This also applies to the judgment of argument
quality, where the assignment of a single ground truth is often questionable.
At the same time, there are generally accepted concepts behind argumentation
that form a common ground. To best represent the interplay of individual and
shared perspectives, we consider a continuum of approaches ranging from models
that fully aggregate perspectives into a majority label to "share
nothing"-architectures in which each annotator is considered in isolation from
all other annotators. In between these extremes, inspired by models used in the
field of recommender systems, we investigate the extent to which architectures
that include layers to model the relations between different annotators are
beneficial for predicting single-annotator labels. By means of two tasks of
argument quality classification (argument concreteness and validity/novelty of
conclusions), we show that recommender architectures increase the averaged
annotator-individual F$_1$-scores up to $43\%$ over a majority label model. Our
findings indicate that approaches to subjectivity can benefit from relating
individual perspectives.