Counterfactual Inference of Second Opinions
This work addresses the challenge of efficiently allocating resources for seeking second opinions in multiclass classification, but it is incremental as it builds on existing causal inference methods.
The paper tackles the problem of designing automated decision support systems to infer second opinions from experts by proposing a counterfactual inference approach, showing that their model improves accuracy over non-causal methods in experiments on synthetic and real data.
Automated decision support systems that are able to infer second opinions from experts can potentially facilitate a more efficient allocation of resources; they can help decide when and from whom to seek a second opinion. In this paper, we look at the design of this type of support systems from the perspective of counterfactual inference. We focus on a multiclass classification setting and first show that, if experts make predictions on their own, the underlying causal mechanism generating their predictions needs to satisfy a desirable set invariant property. Further, we show that, for any causal mechanism satisfying this property, there exists an equivalent mechanism where the predictions by each expert are generated by independent sub-mechanisms governed by a common noise. This motivates the design of a set invariant Gumbel-Max structural causal model where the structure of the noise governing the sub-mechanisms underpinning the model depends on an intuitive notion of similarity between experts which can be estimated from data. Experiments on both synthetic and real data show that our model can be used to infer second opinions more accurately than its non-causal counterpart.