Even Small Correlation and Diversity Shifts Pose Dataset-Bias Issues
This work addresses dataset bias issues for researchers and practitioners in machine learning, providing insights into model behavior under distribution shifts, but it is incremental as it builds on existing shift analysis protocols.
The paper tackled the problem of distribution shifts in deep learning models by analyzing diversity and correlation shifts, finding that models propagate correlation shifts even with low-bias training and that diversity shifts can reduce reliance on spurious correlations, with results demonstrated on a skin cancer classification task using out-of-distribution datasets.
Distribution shifts are common in real-world datasets and can affect the performance and reliability of deep learning models. In this paper, we study two types of distribution shifts: diversity shifts, which occur when test samples exhibit patterns unseen during training, and correlation shifts, which occur when test data present a different correlation between seen invariant and spurious features. We propose an integrated protocol to analyze both types of shifts using datasets where they co-exist in a controllable manner. Finally, we apply our approach to a real-world classification problem of skin cancer analysis, using out-of-distribution datasets and specialized bias annotations. Our protocol reveals three findings: 1) Models learn and propagate correlation shifts even with low-bias training; this poses a risk of accumulating and combining unaccountable weak biases; 2) Models learn robust features in high- and low-bias scenarios but use spurious ones if test samples have them; this suggests that spurious correlations do not impair the learning of robust features; 3) Diversity shift can reduce the reliance on spurious correlations; this is counter intuitive since we expect biased models to depend more on biases when invariant features are missing. Our work has implications for distribution shift research and practice, providing new insights into how models learn and rely on spurious correlations under different types of shifts.