It HAS to be Subjective: Human Annotator Simulation via Zero-shot
  Density Estimation
Wen Wu Wenlin Chen Chao Zhang Philip C. Woodland
Human annotator simulation (HAS) serves as a cost-effective substitute for
human evaluation such as data annotation and system assessment. Human
perception and behaviour during human evaluation exhibit inherent variability
due to diverse cognitive processes and subjective interpretations, which should
be taken into account in modelling to better mimic the way people perceive and
interact with the world. This paper introduces a novel meta-learning framework
that treats HAS as a zero-shot density estimation problem, which incorporates
human variability and allows for the efficient generation of human-like
annotations for unlabelled test inputs. Under this framework, we propose two
new model classes, conditional integer flows and conditional softmax flows, to
account for ordinal and categorical annotations, respectively. The proposed
method is evaluated on three real-world human evaluation tasks and shows
superior capability and efficiency to predict the aggregated behaviours of
human annotators, match the distribution of human annotations, and simulate the
inter-annotator disagreements.