SelecMix: Debiased Learning by Contradicting-pair Sampling
This addresses the issue of biased training data in machine learning, particularly for applications where spurious correlations can lead to unfair or incorrect predictions, representing an incremental improvement over existing debiasing methods.
The paper tackles the problem of neural networks learning unintended decision rules from biased training data by proposing SelecMix, a method that applies mixup to contradicting pairs of examples to debias learning, achieving effectiveness on standard benchmarks, especially in scenarios with label noise.
Neural networks trained with ERM (empirical risk minimization) sometimes learn unintended decision rules, in particular when their training data is biased, i.e., when training labels are strongly correlated with undesirable features. To prevent a network from learning such features, recent methods augment training data such that examples displaying spurious correlations (i.e., bias-aligned examples) become a minority, whereas the other, bias-conflicting examples become prevalent. However, these approaches are sometimes difficult to train and scale to real-world data because they rely on generative models or disentangled representations. We propose an alternative based on mixup, a popular augmentation that creates convex combinations of training examples. Our method, coined SelecMix, applies mixup to contradicting pairs of examples, defined as showing either (i) the same label but dissimilar biased features, or (ii) different labels but similar biased features. Identifying such pairs requires comparing examples with respect to unknown biased features. For this, we utilize an auxiliary contrastive model with the popular heuristic that biased features are learned preferentially during training. Experiments on standard benchmarks demonstrate the effectiveness of the method, in particular when label noise complicates the identification of bias-conflicting examples.