Improving Robustness to Multiple Spurious Correlations by Multi-Objective Optimization
This work addresses the problem of robust model training against multiple spurious correlations for machine learning practitioners, representing an incremental advance by extending debiasing techniques to handle multiple biases simultaneously.
The paper tackles the challenge of training unbiased models when datasets contain multiple biases, which can create conflicting shortcuts during training, and proposes a multi-objective optimization method that dynamically adjusts group-wise losses to achieve a minimax Pareto solution, achieving state-of-the-art results on three multi-bias datasets and superior performance on single-bias datasets.
We study the problem of training an unbiased and accurate model given a dataset with multiple biases. This problem is challenging since the multiple biases cause multiple undesirable shortcuts during training, and even worse, mitigating one may exacerbate the other. We propose a novel training method to tackle this challenge. Our method first groups training data so that different groups induce different shortcuts, and then optimizes a linear combination of group-wise losses while adjusting their weights dynamically to alleviate conflicts between the groups in performance; this approach, rooted in the multi-objective optimization theory, encourages to achieve the minimax Pareto solution. We also present a new benchmark with multiple biases, dubbed MultiCelebA, for evaluating debiased training methods under realistic and challenging scenarios. Our method achieved the best on three datasets with multiple biases, and also showed superior performance on conventional single-bias datasets.