Coherent Hierarchical Multi-Label Learning to Defer for Medical Imaging
For medical imaging workflows with hierarchical label taxonomies, this work solves a critical coherence problem in deferral decisions that prior L2D methods ignored.
This paper introduces the first Learning to Defer (L2D) framework for hierarchical multi-label decisions in medical imaging, addressing deferral incoherence such as taxonomic contradictions and delegation violations. The proposed methods, exact coherent projection and Taxonomic Belief Propagation with Recursive Policy Optimisation, reduce incoherence to near zero while maintaining strong utility.
Learning to Defer (L2D) enables a model to predict autonomously or defer to an expert, but prior work largely assumes flat label spaces. We study the first L2D setting with hierarchical multi-label decisions, motivated by medical-imaging workflows in which findings are organised by clinical taxonomies. In this setting, deferral is a delegation action rather than a label assignment, so treating it as an independent per-label decision can produce deferral incoherence, including taxonomic contradictions, delegation violations, and deferrals of labels already implied by the model's own assertions. We formalise coherent hierarchical deferral under a Selective-Exclusion handoff contract, characterise the Bayes-optimal coherent deferral rule, and show that even nodewise Bayes L2D can be action-incoherent. We then propose two remedies: exact coherent projection, a dynamic-programming decoder over the coherent action set, and Taxonomic Belief Propagation (TBP) with Recursive Policy Optimisation (RPO), a contract-aware joint action model trained through the same recursion used at inference. Across real-reader and controlled-expert medical-imaging benchmarks, naive binary-relevance L2D exhibits non-trivial incoherence. Projection removes it exactly, and fast TBP+RPO drives incoherence near zero while retaining strong utility.