Ranking a Set of Objects using Heterogeneous Workers: QUITE an Easy
  Problem
Alessandro Nordio Alberto tarable Emilio Leonardi
We focus on the problem of ranking $N$ objects starting from a set of noisy
pairwise comparisons provided by a crowd of unequal workers, each worker being
characterized by a specific degree of reliability, which reflects her ability
to rank pairs of objects. More specifically, we assume that objects are endowed
with intrinsic qualities and that the probability with which an object is
preferred to another depends both on the difference between the qualities of
the two competitors and on the reliability of the worker. We propose QUITE, a
non-adaptive ranking algorithm that jointly estimates workers' reliabilities
and qualities of objects. Performance of QUITE is compared in different
scenarios against previously proposed algorithms. Finally, we show how QUITE
can be naturally made adaptive.