No Need for Learning to Defer? A Training Free Deferral Framework to Multiple Experts through Conformal Prediction
This provides a scalable, retraining-free alternative to Learning to Defer for real-world human-AI collaboration, addressing sensitivity to expert changes.
The paper tackles the problem of hybrid human-AI decision-making by proposing a training-free deferral framework that uses conformal prediction to select experts based on label-specific uncertainty, achieving accuracies up to 99.57% on datasets like CIFAR10-H and reducing expert workload by up to a factor of 11.
AI systems often fail to deliver reliable predictions across all inputs, prompting the need for hybrid human-AI decision-making. Existing Learning to Defer (L2D) approaches address this by training deferral models, but these are sensitive to changes in expert composition and require significant retraining if experts change. We propose a training-free, model- and expert-agnostic framework for expert deferral based on conformal prediction. Our method uses the prediction set generated by a conformal predictor to identify label-specific uncertainty and selects the most discriminative expert using a segregativity criterion, measuring how well an expert distinguishes between the remaining plausible labels. Experiments on CIFAR10-H and ImageNet16-H show that our method consistently outperforms both the standalone model and the strongest expert, with accuracies attaining $99.57\pm0.10\%$ and $99.40\pm0.52\%$, while reducing expert workload by up to a factor of $11$. The method remains robust under degraded expert performance and shows a gradual performance drop in low-information settings. These results suggest a scalable, retraining-free alternative to L2D for real-world human-AI collaboration.