Designing Closed-Loop Models for Task Allocation
Vijay Keswani L. Elisa Celis Krishnaram Kenthapadi Matthew Lease
Automatically assigning tasks to people is challenging because human
performance can vary across tasks for many reasons. This challenge is further
compounded in real-life settings in which no oracle exists to assess the
quality of human decisions and task assignments made. Instead, we find
ourselves in a "closed" decision-making loop in which the same fallible human
decisions we rely on in practice must also be used to guide task allocation.
How can imperfect and potentially biased human decisions train an accurate
allocation model? Our key insight is to exploit weak prior information on
human-task similarity to bootstrap model training. We show that the use of such
a weak prior can improve task allocation accuracy, even when human
decision-makers are fallible and biased. We present both theoretical analysis
and empirical evaluation over synthetic data and a social media toxicity
detection task. Results demonstrate the efficacy of our approach.