Causally motivated Shortcut Removal Using Auxiliary Labels
This work addresses the problem of robust machine learning for researchers and practitioners by providing a flexible approach to mitigate shortcut learning, though it builds on existing causal inference techniques.
The paper tackles shortcut learning by proposing a causally-motivated regularization method using auxiliary labels to discourage models from relying on unstable associations, resulting in more robust estimators with better generalization under distribution shift and improved finite sample efficiency.
Shortcut learning, in which models make use of easy-to-represent but unstable associations, is a major failure mode for robust machine learning. We study a flexible, causally-motivated approach to training robust predictors by discouraging the use of specific shortcuts, focusing on a common setting where a robust predictor could achieve optimal \emph{iid} generalization in principle, but is overshadowed by a shortcut predictor in practice. Our approach uses auxiliary labels, typically available at training time, to enforce conditional independences implied by the causal graph. We show both theoretically and empirically that causally-motivated regularization schemes (a) lead to more robust estimators that generalize well under distribution shift, and (b) have better finite sample efficiency compared to usual regularization schemes, even when no shortcut is present. Our analysis highlights important theoretical properties of training techniques commonly used in the causal inference, fairness, and disentanglement literatures. Our code is available at https://github.com/mymakar/causally_motivated_shortcut_removal