Causality-aware counterfactual confounding adjustment for feature representations learned by deep models
This work addresses confounding issues in deep learning for researchers and practitioners, but it is incremental as it extends existing linear methods to DNNs.
The paper tackled the problem of confounding in deep neural network feature representations by adapting a linear counterfactual deconfounding method to DNNs, showing it effectively combats confounding and improves model stability on colored MNIST datasets under dataset shifts.
Causal modeling has been recognized as a potential solution to many challenging problems in machine learning (ML). Here, we describe how a recently proposed counterfactual approach developed to deconfound linear structural causal models can still be used to deconfound the feature representations learned by deep neural network (DNN) models. The key insight is that by training an accurate DNN using softmax activation at the classification layer, and then adopting the representation learned by the last layer prior to the output layer as our features, we have that, by construction, the learned features will fit well a (multi-class) logistic regression model, and will be linearly associated with the labels. As a consequence, deconfounding approaches based on simple linear models can be used to deconfound the feature representations learned by DNNs. We validate the proposed methodology using colored versions of the MNIST dataset. Our results illustrate how the approach can effectively combat confounding and improve model stability in the context of dataset shifts generated by selection biases.