Learning What Makes a Difference from Counterfactual Examples and Gradient Supervision
It addresses generalization failures in deep learning for computer vision and NLP, though it is incremental as it builds on existing datasets and methods.
The paper tackles the problem of deep learning models learning spurious correlations by proposing an auxiliary training objective that uses counterfactual example pairs to improve generalization, showing improved performance on out-of-distribution test sets in vision and NLP tasks.
One of the primary challenges limiting the applicability of deep learning is its susceptibility to learning spurious correlations rather than the underlying mechanisms of the task of interest. The resulting failure to generalise cannot be addressed by simply using more data from the same distribution. We propose an auxiliary training objective that improves the generalization capabilities of neural networks by leveraging an overlooked supervisory signal found in existing datasets. We use pairs of minimally-different examples with different labels, a.k.a counterfactual or contrasting examples, which provide a signal indicative of the underlying causal structure of the task. We show that such pairs can be identified in a number of existing datasets in computer vision (visual question answering, multi-label image classification) and natural language processing (sentiment analysis, natural language inference). The new training objective orients the gradient of a model's decision function with pairs of counterfactual examples. Models trained with this technique demonstrate improved performance on out-of-distribution test sets.