Spurious Correlation-Aware Embedding Regularization for Worst-Group Robustness
This addresses the issue of model robustness for underrepresented groups in subpopulation shift scenarios, offering a novel theoretical and practical approach to mitigate spurious correlations.
The paper tackles the problem of deep learning models relying on spurious correlations, which makes them vulnerable to distribution shifts, especially in underrepresented groups, by proposing Spurious Correlation-Aware Embedding Regularization (SCER) to suppress spurious cues and improve worst-group robustness, showing that SCER outperforms prior state-of-the-art methods in worst-group accuracy.
Deep learning models achieve strong performance across various domains but often rely on spurious correlations, making them vulnerable to distribution shifts. This issue is particularly severe in subpopulation shift scenarios, where models struggle in underrepresented groups. While existing methods have made progress in mitigating this issue, their performance gains are still constrained. They lack a rigorous theoretical framework connecting the embedding space representations with worst-group error. To address this limitation, we propose Spurious Correlation-Aware Embedding Regularization for Worst-Group Robustness (SCER), a novel approach that directly regularizes feature representations to suppress spurious cues. We show theoretically that worst-group error is influenced by how strongly the classifier relies on spurious versus core directions, identified from differences in group-wise mean embeddings across domains and classes. By imposing theoretical constraints at the embedding level, SCER encourages models to focus on core features while reducing sensitivity to spurious patterns. Through systematic evaluation on multiple vision and language, we show that SCER outperforms prior state-of-the-art studies in worst-group accuracy. Our code is available at \href{https://github.com/MLAI-Yonsei/SCER}{https://github.com/MLAI-Yonsei/SCER}.