Designing Closed-Loop Models for Task Allocation
This addresses the problem of automated task allocation in real-world settings without ground truth, but it is incremental as it builds on existing methods with a novel bootstrapping approach.
The paper tackles the challenge of training accurate task allocation models using imperfect human decisions by exploiting weak prior information on human-task similarity, showing improved allocation accuracy even with fallible and biased decision-makers.
Automatically assigning tasks to people is challenging because human performance can vary across tasks for many reasons. This challenge is further compounded in real-life settings in which no oracle exists to assess the quality of human decisions and task assignments made. Instead, we find ourselves in a "closed" decision-making loop in which the same fallible human decisions we rely on in practice must also be used to guide task allocation. How can imperfect and potentially biased human decisions train an accurate allocation model? Our key insight is to exploit weak prior information on human-task similarity to bootstrap model training. We show that the use of such a weak prior can improve task allocation accuracy, even when human decision-makers are fallible and biased. We present both theoretical analysis and empirical evaluation over synthetic data and a social media toxicity detection task. Results demonstrate the efficacy of our approach.