Consistent Estimators for Learning to Defer to an Expert
This addresses the problem of integrating machine learning with expert decision-making in practical scenarios, offering a method to optimize when to defer, which is incremental as it builds on existing cost-sensitive learning frameworks.
The paper tackles the problem of learning predictors that can either make a prediction or defer to an expert, given only samples of the expert's decisions, by proposing a procedure based on learning a classifier and a rejector with theoretical analysis. The result is a novel reduction to cost-sensitive learning with a consistent surrogate loss that generalizes cross-entropy, shown effective in various experimental tasks.
Learning algorithms are often used in conjunction with expert decision makers in practical scenarios, however this fact is largely ignored when designing these algorithms. In this paper we explore how to learn predictors that can either predict or choose to defer the decision to a downstream expert. Given only samples of the expert's decisions, we give a procedure based on learning a classifier and a rejector and analyze it theoretically. Our approach is based on a novel reduction to cost sensitive learning where we give a consistent surrogate loss for cost sensitive learning that generalizes the cross entropy loss. We show the effectiveness of our approach on a variety of experimental tasks.