Expert-Agnostic Learning to Defer
This addresses the challenge of generalizing to unseen expert behaviors in autonomous systems, particularly for medical imaging, and is incremental as it builds on prior learning-to-defer methods.
The paper tackled the problem of learning to defer to human experts by introducing Expert-Agnostic Learning to Defer (EA-L2D), which uses a Bayesian approach to model expert behavior agnostically, resulting in up to a 28% relative improvement on unseen experts across medical imaging datasets.
Learning to Defer (L2D) trains autonomous systems to handle straightforward cases while deferring uncertain ones to human experts. Recent advancements in this field have introduced methods that offer flexibility to unseen experts at test time. However, we find these approaches struggle to generalise to experts with behaviours not seen during training, require extensive human annotation, and lack mechanisms for incorporating prior knowledge of expert capabilities. To address these challenges, we introduce Expert-Agnostic Learning to Defer (EA-L2D), a novel L2D framework that employs a Bayesian approach to model expert behaviour in an \textit{expert-agnostic} fashion. Across benchmark medical imaging datasets (HAM10000, Blood Cells, Retinal OCT, and Liver Tumours), EA-L2D significantly outperforms prior methods on unseen experts, achieving up to a 28\% relative improvement, while also matching or exceeding state-of-the-art performance on seen experts.