DivIL: Unveiling and Addressing Over-Invariance for Out-of- Distribution Generalization
This addresses a key limitation in invariant learning methods for improving model robustness in out-of-distribution scenarios, though it is incremental as it builds on existing IL approaches.
The paper tackles the problem of over-invariance in invariant learning for out-of-distribution generalization, where strong constraints can degrade performance by losing important invariant features, and proposes DivIL to address this, achieving improved results across 12 datasets and 6 models.
Out-of-distribution generalization is a common problem that expects the model to perform well in the different distributions even far from the train data. A popular approach to addressing this issue is invariant learning (IL), in which the model is compiled to focus on invariant features instead of spurious features by adding strong constraints during training. However, there are some potential pitfalls of strong invariant constraints. Due to the limited number of diverse environments and over-regularization in the feature space, it may lead to a loss of important details in the invariant features while alleviating the spurious correlations, namely the over-invariance, which can also degrade the generalization performance. We theoretically define the over-invariance and observe that this issue occurs in various classic IL methods. To alleviate this issue, we propose a simple approach Diverse Invariant Learning (DivIL) by adding the unsupervised contrastive learning and the random masking mechanism compensatory for the invariant constraints, which can be applied to various IL methods. Furthermore, we conduct experiments across multiple modalities across 12 datasets and 6 classic models, verifying our over-invariance insight and the effectiveness of our DivIL framework. Our code is available at https://github.com/kokolerk/DivIL.