Mitigating Shortcut Learning with Diffusion Counterfactuals and Diverse Ensembles
This addresses the problem of biased model predictions in machine learning, particularly in computer vision, by mitigating shortcut learning without costly data collection, though it is incremental as it builds on existing ensemble and diffusion model techniques.
The paper tackles shortcut learning caused by spurious correlations in data by proposing DiffDiv, a framework using diffusion models to generate synthetic counterfactuals for ensemble diversification, which removes dependence on shortcut cues without extra supervision and shows improved generalization comparable to methods requiring additional data.
Spurious correlations in the data, where multiple cues are predictive of the target labels, often lead to a phenomenon known as shortcut learning, where a model relies on erroneous, easy-to-learn cues while ignoring reliable ones. In this work, we propose DiffDiv an ensemble diversification framework exploiting Diffusion Probabilistic Models (DPMs) to mitigate this form of bias. We show that at particular training intervals, DPMs can generate images with novel feature combinations, even when trained on samples displaying correlated input features. We leverage this crucial property to generate synthetic counterfactuals to increase model diversity via ensemble disagreement. We show that DPM-guided diversification is sufficient to remove dependence on shortcut cues, without a need for additional supervised signals. We further empirically quantify its efficacy on several diversification objectives, and finally show improved generalization and diversification on par with prior work that relies on auxiliary data collection.