On the Connection between Invariant Learning and Adversarial Training for Out-of-Distribution Generalization
This addresses the issue of model robustness for AI systems in real-world scenarios, but it is incremental as it builds on existing methods like IRM and AT.
The paper tackles the problem of deep learning models failing on out-of-distribution data due to reliance on spurious features, and proposes Domainwise Adversarial Training (DAT), which improves OOD generalization under both correlation and diversity shifts.
Despite impressive success in many tasks, deep learning models are shown to rely on spurious features, which will catastrophically fail when generalized to out-of-distribution (OOD) data. Invariant Risk Minimization (IRM) is proposed to alleviate this issue by extracting domain-invariant features for OOD generalization. Nevertheless, recent work shows that IRM is only effective for a certain type of distribution shift (e.g., correlation shift) while it fails for other cases (e.g., diversity shift). Meanwhile, another thread of method, Adversarial Training (AT), has shown better domain transfer performance, suggesting that it has the potential to be an effective candidate for extracting domain-invariant features. This paper investigates this possibility by exploring the similarity between the IRM and AT objectives. Inspired by this connection, we propose Domainwise Adversarial Training (DAT), an AT-inspired method for alleviating distribution shift by domain-specific perturbations. Extensive experiments show that our proposed DAT can effectively remove domain-varying features and improve OOD generalization under both correlation shift and diversity shift.