Spot Check Equivalence: an Interpretable Metric for Information
  Elicitation Mechanisms
Shengwei Xu Yichi Zhang Paul Resnick Grant Schoenebeck
Because high-quality data is like oxygen for AI systems, effectively
eliciting information from crowdsourcing workers has become a first-order
problem for developing high-performance machine learning algorithms. Two
prevalent paradigms, spot-checking and peer prediction, enable the design of
mechanisms to evaluate and incentivize high-quality data from human labelers.
So far, at least three metrics have been proposed to compare the performances
of these techniques [33, 8, 3]. However, different metrics lead to divergent
and even contradictory results in various contexts. In this paper, we harmonize
these divergent stories, showing that two of these metrics are actually the
same within certain contexts and explain the divergence of the third. Moreover,
we unify these different contexts by introducing \textit{Spot Check
Equivalence}, which offers an interpretable metric for the effectiveness of a
peer prediction mechanism. Finally, we present two approaches to compute spot
check equivalence in various contexts, where simulation results verify the
effectiveness of our proposed metric.